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ABSTRACT 


During January 1979 VP-31, the West Coast P-3 Fleet 
Replacement Squadron, implemented an Instructional System 
Development based training program to replace their 
traditional training program. Due to monetary, manpower, and 
time constraints, the evaluation phase of the new training 
program was not completely developed or implemented. This 
thesis examines the current status of the external evaluation 
portion of the new training program in an attempt to 
determine the feasibility of its completion and 
implementation. The external evaluation plan is related to 
the Interservice Procedures for Instructional System 
Development Model. From this analysis, a better 
understanding of the plan is gained and recommendations for 
an improved external evaluation program and training system 


are presented. 
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I. INTRODUCTION 


During the last few decades, concentrated efforts have 
been directed toward improving the design of Navy training 
programs. One of the most significant achievements evolving 
from exhaustive research conducted in management sciences, 
communication sciences, and behavioral sciences has been the 
application of systematic procedures to instructional system 
design. The Navy began applying the systematic process to 
the design of training programs in the late LO50t%s:. 
Continued refinement of the initially crude systematic proce- 
dures eventually led to the modern day state-of-the-art “sys- 
tems approach" methodology for design of training systems. 

The systems approach as utilized in instructional design 
1s a process of planning instruction which makes use of 
research and learning theory and employs empirical testing 
data as a means for the improvement of the designed 
Myseenietion {Ref. 1]. Systems approach technology was the 
impetus for the Instructional Systems Development (ISD) 
program which is the approach now required by the Navy, Air 
Force, Army, and Marines for instructional system design. 
Instructional systems development is a systematic planning 
and development process which is designed to optimize 
training effectiveness and efficiency. Theoretically it 


seems to be a far better method to design training programs 








than methods previously employed. This is especially true in 
the area of quality control, because unlike traditional 
design methods, the ISD approach places tools within a 
training system to identify and evaluate its design errcrs. 
This procedure is especially important when the system is 
large and the training complex. 

Since January of 1979, Patrol Squadron Thirty-one (VP- 
Ba), which is the Navy's West Coast P-3 Orion Fleet 
Replacement Squadron (FRS), has been utilizing the ISD 
process to train flight crews for the P-3 aircraft. This 
process was also adopted by Patrol Squadron Thirty (VP-30), 
the East Coast FRS in January of 1980. Development of the P-3 
Instructional System, the VP FRS version of the ISD was 
accomplished over several years by a large number of Navy and 
contractor personnel. The system consists of a series of 
orderly, logical, and interrelated steps to produce a 
training course which is efficient and effective in providing 
graduates with the skills, Knowledge, and attitudes essential 
to the performance of a job [Ref. 2]. 

Since the introduction of the P-3 Instructional System in 
1979, the quality control phase has never been formally 
implemented due to monetary and manpower constraints. These 
undesirable limitations are common to many situations in 
times of economic uncertainty. The lack of an active 


evaluation phase changes the structure of the system from = an 
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LSD process to an exXtremely sophisticated and rigidly 
Structured training program with no means available for 
rigorous evaluation of the system. Therefcre, even though 
the system may be well designed and expertly managed, there 
1s no feasible way to determine whether or not the 
instructional system objectives and required Fleet 
performance levels are being met. The purpose of this thesis 
is to examine the status of the designed quality control 
program and provide recommendations for its completion and 
incorporation. As a result of this examination, an improved 


P-3 FRS training system should be realized. 


A. BACKGROUND 

Prior to the implementation of ISD training procedures, 
VP fleet replacement squadrons utilized traditional training 
methods and media, consisting primarily of the group-paced 
lecture/demonstration concepts. These concepts, unlike ISD, 
offered no affordable means, of locating training design 
errors, to ensure the existance of an optimal instructional 
system. This section will discuss the traditional VP FRS 
training program pointing out some of the serious 
shortcomings that led to the need for a more sophisticated 
systems oriented training process. A brief overview of ISD 


will then be presented, to demonstrate how most traditional 
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training system problems can be solved by using the closed- 


loop “systems approach" for training system design. 


1. ie VP Fleet Replacement squadron Tradi taonaud. 
Training Program 

The traditional training system used by the VP Fleet 
Replacement Squadrons until the late 1970's, was well 
designed and skillfully administered. However, inherent in 
the design methods of the program was the problem of 
measuring training effectiveness. Training managers were at 
best able to make only broad assessments of the activities' 
training value, and their methods were not systematic, 
comprehensive, objective, nor scientifically accurate. 
Therefore this type of evaluation proved to be of no 
Particular value to either training personnel or command 
level managers [Ref. 3]. 

Evaluation of a training program is a critical 
concept needed to determine if the objectives of the program 
and needs of the organization are being met. Also, it must 
determine if the program is in fact providing the necessary 
information for graduates to do their job. Evaluation 
overrides the human error inevitable in the application of 
subjective opinion in providing a means for training activity 
personnel to apraise themselves, their practices and 


products. The evaluation must be specific, continuous, 
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cooperative, conducted in terms of purposes, and based on 
objective methods and standards [Ref. 3]. 

RecogniZing the need to determine his training 
activities’ effectiveness and efficiency, the commanding 
officer of Patrol Squadron Thirty-one requested assistance 
from the Naval Training Device Center (NTDC) in September 
por. The Naval Training Device Center designed an 
evaluation program encompassing the total training function 
of the FRS. The program was modeled after similiar studies 
of other naval aviation training programs. The results of 
the evaluation, known as the "VP-31 Educational Self-Audit" 
sought to. determine the quality of specific aspects of the 
VP-31 training program and to encourage the training staff 
and faculties to develop improved instructional systems and 
materials. 

The evaluation program, based upon systematic and 
detailed procedure, was divided into six related segments; 
(1) Philosophy and Goals, (2) Buildings and Facilities, (3) 
Management, (4) Staff and Faculty, (5) Curriculum, and (6) 
Instructional Support. The extensive analysis of each 
segment identified several basic training discrepancies 
within the FRS program. The evaluation team concluded that 
the most important problems would be corrected and training 
at VP-31 would be significantly improved if the following 


changes were made: (1) Provide professional educational 


ae 





assistance ona full-time basis, (2) provide more complete 
dialogue among all levels of the Air Antisubmarine Warfare 
(AASW) community, (3) improve overall instructional skills of 
instructors and instructor supervisors, (4) implement a 
program leading to increased use of self-instructional 
materials and media, (5) establish revision procedures to 
Stabilize the curriculum, (6) decrease the rate of instructor 
turn-over, and (7) establish better communication between VP- 
31 and all segments of the AASW community. 

The above problems, found by the NTDC team, coupled 
with mumerous other shortcomings such as no clearly written 
Statements Cf philosophy and goals, no statement of 
management objectives, etc., led to the general conclusion 
that a better training system was needed at VP-3l. This 
decision was strongly reinforced in 1974 when it became 
apparent that the traditional training program could not 
possibly handle the upcoming incorporation of the 2F87F 
flight simulator or withstand a proposed twenty-three percent 
reduction in aircraft and manpower. The Commanding Officer, 
Patrol Squadron Thirty-one [Ref. 4] in submitting a proposal 
for instructional development of P-3 FRS training stated: 

"...- This effort will require the complete restructuring 
of the present training program both in concept and format. 
A revision of such magnitude requires a formal development 
program. Unfortunately, major development efforts in the 
Dast employed non-scientific methods and resulted in a R/S 
(Readiness Squadron) training procram of questionable 


efficiency which left the R/S extremely vulnerable to 
reductions during periods of fiscal/personnel austerity. 
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If the new R/S training program is to teach the knowledge 
and skills essential for successful fleet performance and 
be supportable on the basis of cost-effectiveness, the 
process by which it is developed must include the 
deliberate, orderly methods of a systems approach to 
PramnEng." 


This was the first step toward application of ISD 
methods for training system development within the VP 
community. 

2. Instructional Systems Development 

The Random House Dictionary defines instruction as 
the art or practice of furnishing knowledge, especially by a 
systematic method. More specifically, it is a set of events 
employed by those whose purpose it is to develop an organized 
plan to facilitate learning. Instruction then, simply 
stated, 1S a means to help people learn. 

Instruction is often applied to an area as an 
Organized way of accomplishing certain goals. This area of 
interest in the most general terms has no fixed boundaries 
and 1S often referred to as a system. System 1S a relative 
term and can be thought of as composites made up of 
interacting constituent parts [Ref. 5]. From these two ideas 
emerges the term instructional system, whose purpose can be 
thought of as furnishing a means to satisfy specified 
objectives to reach any outcome required by a learning 
program. According to AFM-50 [Ref. 6], the Air Force manual 
for instructional system development, an instructional system 


can be thought of as an integrated combination of resources 
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(students, instructors, materials, equipment, and 
Pact lities), techniques, and procedures performing 
efficiently the functions required to achieve specified 
learning objectives. For the purpose of this discussion, an 
instructional System will be considered a device PCr 
attaining whatever learned outcomes a training program 
intends. 

Instructional design can be a very complex and time- 
consuming process. The fact that there are numerous ways to 
approach this seemingly impossible task only serves’ to 
complicate matters. Gagne' and Briggs [Ref. 1], two pioneers 
in learning concepts and instructional design, present one 
method of design that they consider feasible and worthwhile. 
The process they describe is based upon the following five 
important assumptions. First, the design must be aimed at 
aiding the learning of the individual. Although learners are 
often assembled into groups, learning occurs within each 
member of a group. Second, instructional design contains 
both immediate and long-range phases. The instructor 
preparing lesson plans several hours before a presentation 
constitutes the immediate phase, while the long-rance 
considerations may consist of a set of lessons covering a 
topic and a set of topics covering a course. The third 
assumption is the belief that systematically designed 


instruction can sreatly affect individual human develooment. 
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This "guiding light" concept ensures that all students are 
given an equal opportunity to use their individual talents to 
the fullest degree. The fourth idea specifically states that 
the design of instruction should be conducted by means of a 
systems approach. This idea will be discussed further in the 
latter part of this section. The fifth and one of the most 
important points is that when designing instruction, one must 
take into account how human beings learn in every phase of 
design. 

The aspects of learning theory which have a direct 
impact on instruction are those which relate to controllable 
events and conditions [Ref. 1]. Continued research in the 
area of human learning has'7 brought to laght more 
sophisticated theories than were previously held. t is now 
known that two categories of factors exist in a learning 
Situation. First are the age-old factors COnGIGuey, 


repetition, and reinforcement that exist in an individual's 


external environment. The second and newest category of 
factors are internal in nature. That is, they originate in 
an individual's memory. Some of the most important internal 
Factors are: factual information, intellectual skills, 


Strategies, motivation, and an attitude of confidence. These 
new State-of-the-art concepts depicted in Figure ee 
emphasizes the fact that, in the design of instruction, both 


the external situation stimulating the learner and _ those 
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capabilities which learners bring with them must be 


considered [Ref. l]. 


Contiguity Repetition Reinforcement 
EXTERNAL (Temporal (Arrangement of 
arrangement contingencies) 
FACTORS of conditions) 





Factual Intellectual Strategies 

INTERNAL Information Skills (Cued or 
(May be (Recalled self-activated, 

FACTORS presented or from prior from prior 

recalled from learning) practice} 


prior learning) 


sOurce: “Ref..1: p., 10 
Figure 1.1. Factors Affecting the Learning Event 

As stated earlier in the Gagne' and Briggs' design 
model assumptions, instructional design should be constructed 
by means of a systems approach. This revolutionary concept 
has proven to be an invaluable tool capable of incorporating 
the high technology innovations developed during the last two 
decades into a cohesive plan for the improvement of training 
effectiveness and efficiency. Rapid development in the area 
of educational technology produced such items as programmed 
instruction, the proficiency test, and computer assisted 
instruction in such a short timeframe, that attempts to 
effectively and economically incorporate all of these ideas 
into any sort of an instructional program was impossible. 


With the timely development of the systems approach, it 
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became possible to incorporate these innovative possibilities 
into workable educational plans [Ref. 5]. 

While systematic problem solving is fairly old, it 
was not until after the Second World War that such concepts 
began being used in the field of education. Following the 
War, individuals who had successfully applied the systems 
approach to the design and implementation of complex military 
equipment returned to the field of education and began using 
this new technique to design instructional systems. 

The systems approach is a method for constructing a 
plan that has the capability to design, implement, and 
evaluate a program that has been constructed to achieve 
certain goals. Gagne' and Briggs [Ref. 1] give the following 
description of the systems approach as utilized in their 
instructional design plan. 

",...The systems approach to instructional design involves 
the carrying out of a number of steps beginning with an 


analysis of needs and goals, and ending with an evaluated 
system of instruction which demonstrably succeeds in 


meeting accepted goals. Decisions in each of the 
individual steps are based upon empirical evidence, to the 
extent that such evidence allows. Bach step leads to 


decisions which become ‘inputs! to the next step, so that 
the whole process is as solidly based as is possible within 
the limits of human reason. Furthermore each step is 
checked against evidence that is ‘fed back' from subsequent 
steps to provide an indication of the validity cf the 
system." 


In practice, the systems approach provides an orderly and 
logical way of reaching pre-set goals. It is not a difficult 


concept in theory, but actually applying the approach to 
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improve the operation and success of a system can prove to be 
both time consuming and complex. Effective utilization of 
the concept requires an individual with a solid understanding 
of the method and an enormous amount of past experience in 
the design of each phase. Two reasons for the limited 
success of the systems approach in the area of training in 
the past are: First, the instructional field is vastly more 
complex than any area where the concept has been used in the 
past, and second, few training system designers have a 
reasonable grasp of the approach. The complexity of the 
system becomes readily apparent if one considers all of the 
alternate solutions available at each step of the process and 
identification of the possible effects of each one. This 
complexity should not discourage usage of the system. Once 
the concepts are mastered and enough practice has been 
obtained in this method of instructional design, it is by far 
the most effective tool we have available today to design 
programs to enhance human learning. 

Klaus [Ref . 5] states that most designers of 
instructional systems agree on the following six steps as the 
principle characteristics of a systems approach =o 
instructional design: (1) recognition and definition of the 
problem in terms of the desired outcome of any system action, 
(2) establishment of the criteria to be used in assessing the 


success of a change, (3) identification of potential 
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solutions to the problem, (4) feasibility testing for each 
promising solution, (5) begin assembling the new system 
adopting one or more of the solutions, and (6) provide 
procedures for improving the system as experience is gained 
and limitations or short falls are discovered. This closed- 
loop systems approach is illustrated in Figure 1.2. Note the 
closed-loop and continuous cycle of the system. The dashed 
arrow represents the ability of the system to insert a 
discovered problem into the loop to institute an improvement 
cycle. The system continually corrects itself everytime a 
problem is discovered. Thus, in theory, the longer a system 
is in operation the better it will become. Although there 
are several slightly varying concepts of what constitutes a 


systems approach, this is one of the most highly reguarded. 


ue define probiem establish critena 
.? % in terms of foe accoptability 
a deswed outcomes of a solution 
cowect data to identify promising 
dentify remaining alternatives to meet 
deficiencies critenon requirements 
select solution davelos protatypes 
and make needed of each aitemative 
system adjustments for evaluation 


Source: Ref.5: p. 179 
Figure 1.2. Steps in tne Systems Approach Cycle 


The Armed Services began full-scale utilization of 


the systems approach to training around 1970, when the Air 
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Force adopted a training development concept called 
Instructional Systems Development (ISD). The ISD process is 
a broad application of the systems approach to training which 
Dy design increases the quality of training and reduces’ the 
amount of time and money necessary to train an individual. A 
common misconception that should be corrected from the 
beginning is the fact that ISD is a process through which 
training alternatives are selected, and should not be 
confused with or put on the same level as tools of 
contemporary training technology, such as slide-tape 
presentations, Drogrammed instruction, computer assisted 
instruction, etc. Instructional Systems Development, ina 
broad sense, is a process which allows for the orderly 
development or change of training programs and ensures cost- 
effective instruction that produces graduates capable of 
acceptable performance on the job [Ref. 6]. Since it is an 
application of the systems approach, practical use of the 
process involves the cyclic concept suggested in Figure 1.2. 
As illustrated, in the figure, the constant revision of past 
steps along with projections of future effects serve to 
produce a much stronger and improved system. Empirical 
testing of the system solidifies the process and provides 
maximum effectiveness. Through this process designers can 
actually set design objectives describing acceptabie evidence 


of successful system operation, and continue the design 
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process until that point of quality has been reached 
[Ref. 1). Instructional systems Development can then be 
thought of as a repetitive process of analysis, design, 
verification, and revision. While reviewing the methodology 
and application of ISD in the armed services, Vineberg and 
Joyner [Ref. 7] characterize the sequence of procedures 
constituting ISD as follows: (1) the rigorous derivation of 
training requirements from job requirements, (2) the 
selection of instructional strategies so that the efficiency 
of training is maximized, and (3) interative trial and 
revision of instruction during development until training 
objectives are met. 

Utilizing the ISD process for instituting or revising 
training within the military services is strongly supported 
by the Department of Defense. Within the military 
establishment there exists several models or variations of 
the model constructed for standardization of the ISD process 
for the armed forces. The standardization model is presented 
im five documents titled Interservice Procedures for 
Instructional System Development (IPISD), and is published by 
the Navy as NAVEDTRA 106A [Ref. 8]. The model was developed 
for the Army and later gained approval from the Interservice 
committee on Instructional Systems Development. It provides 
detailed procedures for conducting ISD. The Army uses this 


exact model, distributed as TRADOC Pamphlet 350-30 [REf. 9] 
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for their ISD purposes, while the Navy has published a 
condensed version of the model for its own use, titled 
Procedures for Instructional Systems Development in NAVEDTRA 
110A [Ref. 10]. 

The Marine Corps uses a greatly reduced version of 
the IPISD model that has been altered to meet Marine Corps' 
documentation requirements. Guidelines can be found in 
Marine Corps Order P151.23B [Ref. 11]. The Air Force model 
varies somewhat from the IPISD model. Their version of the 
ISD process can be found in Air Force Manual (AFM) 50-2, 
Instructional System Development [Ref. 6] and Air Force 
Pamphlet (AFP) 50-58, Handbook for Designers of Instructional 
Systems [Ref. 12]. AFM 50-2 provides an overview of the Air 
Force's ISD model, while AFP 50-58 gives detailed procedures 
for implementing the concepts in AFM 50-2. 

Since this thesis deals with Navy training, the focus 
of this discussion will be directed towards the IPISD model. 
As stated in the letter of Promulgation for NAVEDTRA I11OA 
[Ref. 10] (the Navy's condensed version of IPISD), the manual 
",... provides guidance for the analysis, design, development, 
implementation, and control of instructional programs under 
the direct cognizance of the Chief of Naval Education = and 
Training (CNET)." In addition the letter states that, "All 
Curricula developed within or for use within the Naval 


Education and Training Command, except submarine training 
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materials specified in CNETINST 1550.14, will be developed in 
accordance with the policy, procedures, and standards 
contained therein." 

The IPISD model contains procedures that are 
primarily concerned with the “how to do it" aspects of ISD. 
The model is based upon nine important assumptions that must 
be fully understood before attempting to use the model for 
instructional program design, revision, or administration of 
an ISD training program. The basic construction of the model 
consists of five phases, each being a separate and distinct 
function. The model is designed to provide a sequential 
relationship of the phases. 

Figure 1.3 depicts a block diagram of the IPISD 
model. Each phase is shown with appropriate steps to be 
accomplished. The five phases of the model, as shown in the 
figure, are labeled Analyze, Design, Develop, Implement, and 
Soncrol . Phase One, Analyze, is the single most important 
step in the ISD process, because it establishes what the 
program will teach. This phase provides the procedures to 
define what the jobs consist of, breaking these down into a 


detailed listing of duties, tasks, and elements necessary to 


perform a specific job. The analysis involves techniques 
such as job interviews, questionnaires, experienced 
professionals using their best judgment, and group 


interviews. 
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Phase Two, Design, uses the job analysis from Phase 
One to begin designing instruction. The form and 
specifications ole training are created and the 
identification and design of the terminal and enabling 
objectives that an individual must achieve in order to 
Satisfactorily complete the program are produced. Phase 
Three, Develop, deals with the actual preparation of 
instruction. This phase specifies the knowledge and 
performance learning activities that will be used, develops 
Curriculum outlines and instructional management plans, and 
selects instructional materials. When these products have 
been validated by using empirical data, Phase Pour can take 
place. 

Phase Four, Implementation, produces the steps 
necessary to implement the instructional plan that was 
developed in Phase Three, while Phase Five, Control, provides 
the procedures for the evaluation of the program to determine 
how effectively it has met the initialy set goals. An 
internal evaluation is conducted as a continuous process’ to 
assess student performance and to determine the effectiveness 
of the program methods and materials. An external evaluation 
determines if the course is training graduates well enough to 


perform, on the job, the tasks they were trained to perform, 
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and if these tasks are the ones required to properly do the 
Job. These evaluations provide the basis for program 
revision. 

As stated in NAVEDTRA 106A [Ref. 8], the design and 
develocment of instruction must follow an adequate needs 
analysis. In other words, when there is a Known measurable 
discrepancy between a situation as it exists and the 
Situation as it ought to be, instructional programs should be 
designed to eliminate that discrepancy. During initial 
applications of the ISD process, one should not set out to 
run a course through the process, but rather utilize the 
process in such manner that the need for a course becomes 
apparent through the use of the process. 

The ISD process is an extremely sophisticated method 
of instructional development and contains several unique 
characteristics. The process uses painstaking measures’ to 
ensure proper selection of what is to be trained. This is 
accomplished through job surveys and experts from the field 
when a job already exists and from engineering data for new 
jobs and systems. Instructional Systems Deve lopment also 
generates and applies alternative training methods’ selected 
from state-of-the-art technology instead of relying on tradi- 
tional methodology. These advanced techniques aid in the 
assurance of optimal training effectiveness, time efficiency, 


and cost. Thus there is no need for course content reduction 
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to optimize a system. Finally, one of the most important 
characteristics of the ISD process is the provision for 
measuring the quality of instruction and the overall program. 
The process uses test data based on absolute standards of 
performance to grade students and check program quality. One 
last point that should be noted abcut ISD methodology is its 
flexibility. When utilizng the process the model can be 
entered at any step, or if appropriate, actions can be taken 
in several steps simultaneously. 

It should be clearly evident that the ISD process can 
be extremely beneficial in the area of instructional design. 
The process can be adapted to any training program including 
on-the-job training. One can use the process to design new 
instructional programs or to change and improve existing 
ones. Since the process has been in use for several years, 
empirical evidence now supports the claims of improved 
effectiveness, time efficiency, and costs with proper 
application of ISD. Although many still consider the process 
an art, it is by far closer to a science of education than 
the other approaches to instructional design that do not 


include measurement of their effects. 


B. THE P-3 INSTRUCTIONAL SYSTEM 
During the latter part of September 1974 the Commanding 
Officer of VP-3l, ina letter addressed to Commander Patrol 


Wings, U.S. Pacific Fleet [Ref. 4], stated that the P-3 ASW 
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community was in serious need of a new training program to 
overcome manpower and monetary shortages and still provide 
quality training to personnel enroute to fleet squadrons. He 
went on to say that the training program should be developed 
by the methods of a systems approach to training. As justi- 
Fication for requesting this particular type of instructional 
system design, he cited the Naval Training Equipment Center's 
(NTECsS) Training Analysis and Evaluation Group (TAEG) report 
number 10 [Ref. 13] which contains the results of an experi- 
mental P-3 pilot and flight engineer training program. The 
experiment demonstrated the efficiency and economy of a 
systems approach to training by indicating a potential media 
and manpower savings of six million dollars per year in just 
that one portion of FRS training. The Commanding Officer of 
VP-31 further stated that as trainers advance in sophistica- 
tion, complexity, and cost, the systems approach to training 
could be used as a tool to ensure that FRS training needs are 
met and at the same time guard against the waste of allocated 
funds. 

As a solution to the FRS training problem, the framework 
FOr an instructional development program employing the 
systems approach was submitted. The program was patterned 
after the S-3 Fleet Instruction Team program, and was 
Gesigned to: (1) teach only the essentials for successful 


job performance in the fleet, (2) employ the latest and best 
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technology in design considerations, and (3) maximize the use 
of available resources (i.e., training aids, aircraft, and 
manpower). 

With processing through normal channels complete and 
approval obtained, Courseware, Inc. was commissioned along 
with Navy and government personnel to provide the expertise 
for the new training program. These developmental personnel 
were formed into an Instructional Systems Development Team 
consisting of subject matter experts (SMEs), instruction 
psychologists (IPs), and technical assistants (TAs). The 
results of their efforts provided training programs for thir- 
teen positions on the P-3 A/B and P=-3C aircraft incorporated 
into eleven volumes. 

The team of experts employed the ISD process for the 
program design, which was, by this time, the method required 
by all uniformed services. Courseware, Inc. concurred with 
this state-of-the-art approach, stating that it offered the 
best opportunity for effective, efficient, and motivating 
instruction. Thus the philosophy underlying the design con- 
cept adopted by Courseware, Inc. was: determine what to 
teach, determine how to teach it, and determine whether or 
not the teaching has been successful. 

The basic systems approach developed and validated by 
Courseware, Inc. is based upen six assumptions that describe 


their philosophy on the when, how, and why of instructional 
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design. The ISD approach they developed for the P-3 Instruc- 
tion system consists of ten major steps which served as_ the 
framework for the project. The steps for the most part follow 
those of the IPISD model presented in the last section, 
therefore they will not be covered in any great detail. 

Step one, job analysis, breaks the job down into compo- 
nent tasks. The level of detail is kept relatively general 
through this’ step. Step two, selection of tasks’ for 
training, identifies those tasks listed in step one that must 
be trained in the program. Step three, hierarchical 
analysis, converts job requirements to a set of specific 
training objectives. This step reduces the tasks of step one 
to a more specific level of detail. These new tasks are 
presented in the form of a task diagram which is used to aid 
SMES. 

Step four, media selection, provides media alternatives 
to fit the learning requirements for each objective. The 
media alternatives selected for the program were workbooks, 
videotape, slide/tape, and simulators. Step five, sequencing 
of objectives, constructs the order in which training 
materials will be presented to the student. Step six, lesson 
Specification, expands each sequenced objective to aid the 
course writers. Step seven, authoring lessons, creates sound 


instruction given a specification of strategy, content, and 
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media. Step eight, production of tests, creates lesson tests 
to measure student learning and course effectiveness. 

Step nine, training, is the actual use of instructional 
materials with students. Step ten, evaluation, uses empiri- 
cal means to determine whether students are learning the 
course material, if students can perform adequately on the 
job, and if the course is presenting the proper information 
for the job to be completed. This step was scheduled to occur 
during the first year of training. These ten steps provided 
the basis for the training system development. 

Coincident with the completion of the work established by 
the ten basic steps of the design system was the production 
of the P-3 Instructional Systems Management Plan. The purpose 
of the plan was to provide explicit guidelines and procedures 
to aid in the implementation of the training materials 
evolving out of the development of the new training program. 
The plan was also devised to maintain the materials in an up- 
to-date status as new systems and procedures arise and as 
revisions are required. 

The management plan covers such SOAS as administrative 
details, scheduling, course materials, equipment, facilities, 
and quality control. Additionally, the plan assigns specific 
organizational responsibilities to VP-3l and to the Fleet 
Aviation Specialized Operational (FASO) Training Group 


Pacific Fleet Detachment Moffett. 
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II. STATEMENT OF PROBLEM 


A. PROBLEM STATEMENT 

The primary mission of the peace-time Navy is to train 
for its role as protector of the seas, in time of aggression 
against the United States (U.S.) or in support of U.S. 
interests. The success of the Navy's overall training system 
is determined by how well the numerous training subsystems 
meet their pre-established goals. The attainment of these 
goals plays a critical part in the bid to reach and maintain 
a high-level of Fleet readiness. Thus, designers of Naval 
instructional systems must constantly strive to create 
programs that are optimally effective, time-efficient, and as 
cost-effective as possible to operate. The only way to 
insure the development of such high-caliber programs is to 
include, within the program, a check-and-balance phase that 
will provide continuous revision both during design and after 
the program becomes operational. 

Since the early 1970's the Chief of Naval Education and 
Training has considered the measurement of training 
effectiveness through evaluation of trainee performance, a 
matter of major concern. Despite this high level of concern, 
the ability to design and successfully incorporate meaningful 
training system appraisal continues to. be a problem 


throughout the Navy and all other branches of the military 
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service. The most common denominator in this problem is, as 
most would expect, a lack of adequate assets to properly 
operate the evaluation phase of instructional systems, even 
after the phase has been designed as an integral part of the 
training program. 

The P-3 Instructional System based on ISD methodology, 
was implemented as the FRS training program at VP-3l in 
January 1979. The system was designed with the intention of 
covering every facet of VP Antisubmarine Warfare Training 
required to make optimum use of the P~3 aircraft. A quality 
control plan was included within the system to validate all 
aspects of the instructional design and to provide the 
necessary means for program revision when appropriate. The 
design of the evaluation plan was unavoidably restricted by 
constraints stemming from inadequate assets. Additionally, 
when the training system was implemented, the available funds 
and manpower were insufficient to allow the inclusion of the 
evaluation phase of the system. Thus, not only was’ the 
evaluation Plan incomplete according to classical ISD 
Standards, but also the plan, as it existed, was never 
formally implemented into the training system. 

Another serious problem plaguing the Quality Control plan 
waS a material/media revision. Shortly after the training 
system was implemented in 1979, it was discovered that due to 


such discrepancies as instructional design errors, technical 
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content changes, and editorial errors, some of the designed 
materials would have to be immediately revised. These 
revisions included a change in format for tests. The 
necessary changes were immediately incorporated into the 
system without being validated. This violated the systematic 
basis for the instructional design process and _ seriously 
jeopardized the evaluation plan, especially in the area of 
internal evaluation. 

The P-3 Instructional System, as it exists today, appears 
to be superior to the traditional system, but the only way 
this can be verified is through an appropriate evaluation 
Plan. Without a procedurally correct evaluation plan in 
effect, the ability to determine whether system objectives 
are being met and graduates are capable of satisfactory 
performance on the job on a continuous basis, is questionable 
at best. The currently used method of evaluation is informal 
in nature, and for the most part, consists of sporadic usage 
of a traditionally constructed student critique/questionnaire 
and word-of-mouth reports at monthly conferences held by 
Commanding Officers of Fleet Squadrons. 

Initial justification for the redesign of VP FRS training 
as stated in Chapter One was to provide a high-quality 
training program that could readilly adapt to new and ad- 
vanced technology while reducing operating costs and manpower 


needs. To meet these requirements a new instructional pro- 
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gram was designed using the ISD process, which is a closed- 
loop system. The evaluation phase of this type of program is 
required to provide systematic maintenance and revision. 
Without proper evaluation, there cannot be a closed-loop 
system, But only a rigidly structured traditional-type 
training program. Theoretically one can say that without an 
evaluation phase, there is no such thing as an ISD program. 

ES correct the Quality Control Plan shortcomings 
mentioned above, a problem that must be addressed is, how can 
a complete evaluation phase be incorporated into the P-3 
Instructional System to regain the benefits of the powerful 
ISD concept under which it was designed? 

The intent of this thesis is to make contributions 
towards a more effective VP FRS training system. 
Recommendations will be presented for the completion and 
incorporation of the P-3 evaluation process, which will aid 


in a better overall training program. 


B. THE IMPORTANCE OF INSTRUCTIONAL SYSTEM EVALUATION 

The undisputed need for measurement of training 
effectiveness has long been realized. Without the capability 
of determining how well the instructional program is being 
designed, if students are meeting design objectives, and 
whether or not students are properly trained for, and are 


capable of, adequately doing their job in the fleet, a 
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training system can only be mediocre at best. According to 
Rail, Rankin, and Aagard [Ref. 14], a major problem plaguing 
traditional training systems, (i.e., non ISD systems) is the 
Getermination of training effectiveness largely by rational 
assessment and intuitions of personnel intimately involved in 
the training process. They further state that this type of 
information tends to be biased and lacks the detail necessary 
for improving specific aspects of training. Therefore, to 
determine training effectiveness and obtain information 
suitable for proper training quality control, one must’ seek 
more objective and systematic means. The ISD process of 
instructional development employs these concepts for training 
system evaluation. 

The remainder of this Chapter will cover the basics of 
training system evaluation followed by an overview of the 
evaluation methods utilized in the IPISD model. Finally, the 
steps utilized in the quality control phase of the P-3 
Instructional System will be discussed. 

Ll. The Nature of Instructional System Evaluation 

Evaluation of an instructional system can best be 
described as ameans of analyzing and interpreting system 
evidence that has been systematicaly gathered. The basis for 
training program evaluation begins with a consideration of 
Criteria and criterion measures. When dealing with training 


programs, two classes of criteria, internal and external 


38 





measures, must be considered. Internal criteria are measures 
concerned with an individual's performance in the training 
situation such as objective exams, and questionnaires 
reflecting trainee attitude changes. On the other hand, 
external criteria are utilized to assess performance on the 
job for which the training program was designed. Measures of 
quantity and quality of production are examples of external 
Criteria. 

According to Landry and Trumbo [Ref. 15], Kirkpatrick 
(Ref. 16] identified four "levels" of criteria for the 
evaluation of training programs: reaction, learning, 
behavior, and results. The reaction and learning levels are 
internal criteria. Reaction criteria deals with how the 
student feels about the training program and learning 
Criteria attempts to measure how much was learned as a result 
of the training. The last two levels are external criteria. 
Behavioral criteria measures performance on the job, while 
results criteria determines the utility of the instruction 
with respect to organizational objectives. 

When building the framework for training system 
evaluation, one must carefully strive for the proper balance 
of internal and external criteria. The careful measurement 
of internal criteria can provide valuable information, but 
these measurements cannot tell what impact the training has 


had on job behaviours or organizational goals. Evaluation of 


39 





the payoff resulting from a training system is determined not 
from what is learned in training but how the individual 
performs on the job. 

While describing the principles of instructional 
Gesign, Gagne" and Briggs [Ref. 1] present three specific 
questions concerning instructional systems that should be 
answered as a result of the designed evaluation process: (1) 
how well have the stated objectives of instruction been met, 
(2) is the new program better than the cne it is replacing, 
and (3) does the new program produce any additional effects? 

Evaluation of topics, courses and instructional 
systems is carried out in two stages. Stage one, formative 
evaluation, is undertaken while the new unit is being 
developed. Its purpose is to provide evidence for use in 
making revisions and improvements. This stage is often 
called system validation. Stage two, summative evaluation, 
is concerned with the measurement of student performance once 
the course or Drogram has been developed. In other words, it 
is primarily concerned with instructional system 
effectiveness. The focus of this report will be on 
summative-type evaluation. 

The basic ISD structured training models used by the 
Armed Services makes provision for summative evaluation to be 
conducted in two parts. Part one, internal evaluation, 1s a 


continuous process des:gned to check instructional 
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effectiveness (i.e., are the students obtaining course 
objectives?). The primary measure of effectiveness is made 
via objective-reference achievement tests. Other measures 
include time to complete lessons, attrition rates, and _ the 
Opinions of instructors and students concerning the training. 
Part two, external evaluation, is designed to measure’ the 
performance of the graduate on the job. The purpose of this 
type of evaluation is to discover deficiencies in a 
gGraduate's ability to perform certain required tasks when 
they reach the Job. Once the causes of the deficiencies are 
determined, revisions to the system can be made to correct 
the flaws. External evaluation also ensures that the 
information obtained from the training program is the 
information needed to do the job. The two most accurate 
methods of external evaluation, testing of graduates on the 
job and direct observation, are usually too SOsSeLy: 
therefore, more often than not evaluators must rely on 
Supervisors' summary evaluation of performance, 
questionnaires and interviews. 

The most important points concerning the conduct of 
an external evaluation are first, they must be extremely 
specific (task level) to isolate inadequate performance, and 
second, it must take place within a few months after the 


Graduate has reached the job so that the skills and knowledge 
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acquired on the job will not influence the information 
received from the training program. 
2. The IPISD Model Evaluation Process 

The Interservice Procedures for Instructional Systems 
Development Manual [Ref. 8] lists course evaluation as_ the 
final phase of instructional design. This important step 
represents the beginning of a perpetual training system 
evaluation process that is intended to make certain that the 
training program continues to be effective and to provide a 
student output quality level that satisfies fleet 
requirements. 

The IPISD model evaluation phase capitalizes on the 
two aspect evaluation processes (internal and external) 
described in the last section. The internal evaluation 
program is used to determine if a course has been 
developed/conducted according to the "standards" specified in 
the ISD procedures. Thus, the internal quality control 
aspect seeks to answer the question, "Is the course teaching 
its stated objectives?". The IPISD model specifically states 
that the effectiveness of both the instruction and the ISD 
process itself will be evaluated. 

The data gathered for the internal evaluation process 
deals primarily with student achievement (i.e., attainment of 
the objectives, as represented by test scores). The model 


also recommends collecting information such as trainee 
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background, entry skills, trainee's evaluation of methods and 
media, time to complete lessons and instructor's evaluation 
of the content. This kind of information will permit 
identification OE weak, ineffective, impractical Or 
unpalatable instruction, and thereby lead to the necessary 
revisions in the instructional program. 

The external evaluation program determines the 
adequacy of the instructional design and development process, 
and trainee proficiency, by a standard external to the 
course. This standard is normally the performance of the 
graduate on the job. The external evaluation process 
considers the questions, can the graduate do the tasks 
required on the job, and has the job changed since it was 
last analyzed? Data gathered for the evaluation may 0e 
obtained from graduates and/or their supervisors. in an 
overview of the IPISD Model external evaluation process, 
NAVEDTRA 106A [Ref. 8] stipulates that information that can 
be obtained from course graduates includes their opinions 
about: how well they believe they can perform on the job, 
training received since arriving on the job, how well the 
course prepared them for the job, the parts of the training 
which were relevant to the job, and the job-related tasks 
which caused them the most problems. Additionally, they 
state that the information that may be gained from 


supervisors includes: how well the graduates can do the job, 
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how those graduates measure up to those who received a 
different form of training, and areas in which the graduates 
have not been properly trained. 

The IPISD model recommends the administration of Job 
Performance Measures (JPMs) in the field 30-90 days after 
graduation as the means of determining whether or not 
graduates can perform their required job tasks. Job 
performance measures are "tests that are used to evaluate 
proficiency of a job holder on each task he performs." [Ref. 
8] The Model states that other methods such as_ interviews 
and questionnaires may be used, but that these techniques 
usually inject a certain amount of bias into the evaluation 
process. 

Determining whether or not a job has changed since it 
was last analyzed is amatter of reviewing the present 
relationship between the Job Performance Measures and actual 
job requizements. Vineberg and Joyner [Ref. 7] describe this 
process as determining the predictive validity of the Job 
Performance Measures. 

Most prominent instructional designers seem to 
consider the administration of Job Performance Measures’ to 
graduates as the most reliable way to measure the adequacy of 
instructional design. This method provides the most accurate 
means of isolating the causes of performance discrepancies 


during the evaluation, but it should be pointed out that it 
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1s also the most expensive to use. The bottom line in 
external evaluation is summed up very explicitly in the IPISD 
model caution: don't revise simply for the sake of revision. 
When the majority of graduates and supervisors are satisfied 
with the quality of training, few, if any, system changes are 
usually needed. 

3. The P-3 Instructional System Evaluation Process 

The final phase of the P-3 Instructional System is 
dedicated to quality control. The purpose of the Quality 
Control program is to ensure that the system is meeting its 
goals in the most optimum manner possible. The program's 
general framework is composed of two primary parts. Part 
one, quality control of materials, aids in the assurance that 
all applicable materials are up-to-date and technically 
COEEeCt. Part two, quality control of the instructional 
system itself, aids in the assurance that the system 15S 
Functioning effectively. 

Generally speaking, the quality control program's 
basic guidelines specifies that material revisions will be 
made when the performance data gathered from device sessions 
indicate a need to revise the written material that teach the 
performance and when data support the existence of a content 
Or instructional deficiency. Data gathered for these 
evaluation purposes may be obtained from instructor or 


Student inputs, analysis of student performance and attitude 
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data, and the Naval Air Training and Operating Procedures 
Standardizaticn (NATOPS) manual update process. 

Part two, quality control of the P-3 Instructional 
System itself, is a continuous process devised to identify 
and resolve problems within different aspects of the system. 
System revisions, to correct problems, are designed to be 
made without deteriorating other parts of the system. Thus, 
any problem should upgrade the entire system and not correct 
one problem at the expense of creating new ones elsewhere in 
the system. Revisions to the system may originate from 
changes in the Master Course Syllabus (the primary scheduling 
tool for the Instructional System), in resource requirements, 
in management procedures, or in course content. SErict 
documentation and distribution of revisions are extremely 
important in this instructional system, since the system is 
large and the two units using it are geographicaly separated. 

This, of course, has been a very broad overview of 
the quality control phase of the P-3 Instructional System 
The entire phase is very detailed and complex as one would 
expect from such a large training program. A more thorough 
discussion providing all of the necessary details will be 


presented in Chapters Three and Five. 
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C. SCOPE OF THESIS 

When one strictly adheres to ISD methodology, internal 
evaluation (1.e., evaluation to determine the effectiveness 
of the performance of an instructional product) and external 
evaluation (i.e., evaluation of the training product in the 
operating environment) are extremely complex tools to design, 
implement and operate. Each procedure of an ISD _ based 
evaluation plan is related to the entire instructional system 
in such a manner that if a nonvalidated change is made in the 
system, the evaluation plan can be seriously degraded. This 
appears to be especially true in the area of internal 
evaluation. For example, the media/materials revision of the 
P-3 Instructional System that took place in 1979 has made it 
necessary for numerous design changes to be made to the 
Quality Control Plan before an accurate internal evaluation 
process can exist. Due to the extent of these required 
design changes, an analysis of the entire P-3 Instructional 
System Quality Control Plan (i.e., internal and external 
evaluation) is beyond the scope of this thesis. Since the 
question that needs an immediate answer (i.e., the bottom 
line in VP training) is, can the VP FRS graduates perform 
satisfactorilly on the job, the scope of this report will be 


limited to an analysis of the external evaluation process. 
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III. P-3 INSTRUCTIONAL SYSTEM EXTERNAL EVALUATION SPECIFICS 


The instructional system that is designed according to an 
ISD model is constructed as a complete cycle. The evaluation 
phase is the key element in that it provides feedback 
necessary to complete the cycle. This feedback is the means 
of ensuring continuing quality in graduate performance. 
Evaluation is that integral part of the training system with 
the capability to locate discrepancies and initiate system 
changes. An evaluation process is not a general item that is 
used to "check" a training system, but rather a specific tool 
that must be designed for each individual system. 

The evaluation process designed by Courseware, Inc., for 
incorporation into the FRS Training System is titled the 
P-3 Instructional System Quality Control Plan. As stated in 
the last chapter, the olan was designed under constraints 
that prevented the development of complete procedures 
necessary for evaluation of the system. The resulting 
shortcomings in the plan and their possible remedies will be 
discussed in Chapter Five of this report. 

Chapter Two presented, in part, a general overview of 
both the IPISD model evaluation procedures and the P-3 
Instructional System evaluation procedures for background 


information. This chapter will focus on the details of 
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evaluating the P-3 Instructional System, especially in the 
area of external evaluation. The material that is presented 
is a summary of the P-3 Instructional System Quality Control 
Plan [Ref. 17] and Management Plan [Ref. 18] and is intended 
to provide the reader with enough detail to follow the 
analysis of the plan in Chapter Five. The plan is presented 
just as it was designed by Courseware, Inc. The first three 
sections of this chapter provide general information that 
pertains to the entire quality control plan (i.e., internal 
and external evaluation), while the remaining two sections 


will cover specifics of the external evaluation plan. 


A. PURPOSE OF THE QUALITY CONTROL PLAN 

Quality control provides the means for instructional 
personnel to systematically ask questions designed to 
evaluate the training program. The devised plan to conduct 
quality control involves three major activities: data 
gathering, analysis and revision. The constraints placed upon 
data gathering produce a chain reaction resulting in similar 
limits being placed upon the level of analysis that can be 
accomplished and finally dictating the number and kinds of 
revisions that can be made to this program. 

Three distinctly separate stages occur in the quality 
Gemtrcol effort. The first stage, validation, takes place 
during the actual development of the instructional program, 


and involves trying out designed material with individual 
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and/or small groups. This stage allows instructional designs 
to guarantee specified results. The second stage, internal 
quality control, is a continuous process that takes place 
during the implementation of the training program. This 
stage provides for the identification and correction of 
deficiencies in the total training program (i.e., both 
management and materials). 

The third stage, external quality control, takes place 
several months after course graduates depart for fleet 
assignments. This process attempts to identify and correct 
problems pertaining to the areas of course relevance to real- 
world Operational requirements. Internal and external 
quality GOntrol must be conducted continuously to 
Satisfactorily maintain the training program. EEguie ws 
details the entire P-3 Instructional System quality control 
process and its outputs. Note the connecting nature of the 


process that links all three stages together. 


B. SCOPE OF THE QUALITY CONTROL PLAN 

This quality control plan was designed to cover the 
personnel and procedures required during the internal and 
external quality control stages. During internal quality 
control, data gathering and analysis will be directed towards 
answering the question, “is the training program effectively 


meeting its defined instructional and performance 
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money, this quality control plan does nct address thr 


(1) Procedures for quality control of the instruc- 
tional management system; this area is covered in Appendix 
I of the P-3 Instructional Svstem Management Plan. 


(2) Complete procedures for the external quality con- 
trol Pian; due to the time requirements for data collection 
amc analysis, the timeframe for completion of the external 
Quality control precess would nave extended bevonc the 
scope obs the contract for development of the P-3 
Instructional System. 

(3) Comtinuous evaluation and revision procedures; 
these procedures are Dest identified toward the end of the 
internal auality control process on the Dasis of = needs 
assessment. The needs assessment and identification o£ 
maintenance ongoing revision procedures were intended to be 
&2 joint Courseware/Navy endeavor based on the #xperience of 
the initial quality control plan. 





C. CONSTRAINTS OF THE QUALITY CONTROL PLAN 

The extent of the evaluation (i.e., the total number and 
Kind of evaluation questions that can be addressed during 
internal and external quality control) is dependent upon 


existing priorities or client constraints. 


1. Design Accomodations 


The quality control plan was custom designed to 
accomodate the following restrictions: 


(1) Fixed Naval personnel resources: this limitation 
affects the amount and kind of data collection activities 
during internal quality control. 


(2) Fixed trainer assets: this limitation imposes 
Strict enforcement of scheduling which limits the time 
available for student/instructor interview data collection. 


(3) Fixed course length: this limitation was appli- 
cable during the implementation year. It didn't allow 
enough flexibility in training time to make up for initial 
instructional or system deficiencies. 


(4) Pixed number of students within a fixed contract 
delivery schedule: this restriction may result in a non- 
representative target population sample size. 
Additionally, initial training program effectiveness may be 
degraded by an insufficient amount of trainer assets. 


(5) Training accountability: during the implementa- 
tion year this restriction forces the emphasis to be placed 
on training rather than data collection for analysis’ and 
revision of the instructional materials. 

(6) Learning environment limitations: lack of readi- 
ness in this area restricted the evaluation of the 
instructional materials and the efficiency of the 
instructional management system. 

2. Revision Priorities 
Based upon the above considerations, several 


revision priorities were established for internal quality 
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Control. Since the primary focus of this report is on the 
external evaluation phase, the priorities will not be 
covered, but may be found in their entirety in the P-3 
Instructional System Quality Control Plan. Revisions made to 
the training program will be made according to the kind and 


amount of data collected. 


D. DESIGN OF THE EXTERNAL QUALITY CONTROL PLAN 

This section of the Quality Control Plan discusses’ the 
general procedures to be used in conducting the development, 
data collection, analysis, and revision phases for external 
evaluation. Further procedural detail was intended to be 
provided through a joint Navy/Courseware endeavor upon 
completion of the second class of students for internal 
Swadity control. 


1. Development Phase 





The purpose of the development phase is to provide 
data collection instruments for use in the data collection 
phase. When considering analysis, the following questions 
apply: 


(1) Can the initial course graduate perform at the 
level expected while on the job? 


(2) Has the initial course graduates' training been 
too extensive in any skill area? 


(3) Has the initial course graduate attained the 


proper Knowledge and skills upon arrival at the operational 
squadron? 
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(4) Does the initial course graduate feel the training 
program adequately prepared him for the real-world job 
tasks? 

Attempts to answer these questions will be conducted 
through two types of data collection instruments. The first 
type, questionnaires, will be sent to squadron supervisors of 
initial course graduates and course graduates themselves. The 
second type, interview questions, will be administered on- 
site to both supervisors and course graduates. 

2. Data Collection Phase 

This phase consists of two major steps: 

(1) Questionnaire data gathering activities. 

(2) On-site interviews with supervisors and initial 
course graduates who have been on the job apprcximately six 
months. 

Questionnaires will be sent to all initial course 
graduates and their job supervisors. For this particular 
quality control plan, four questionnaires currently exist. 
Appendix A shows a sample initial course graduate follow-up 
questionnaire, while Appendix B shows a specific sample 
course graduate questionnaire for the pilot student. 
Appendix C displays a sample job supervisor follow-up 
questionnaire and Appendix D shows a specific sample job 
supervisor follow-up questionnaire for a pilot supervisor. 
As specified in the Management Plan, follow-up questionnaires 


will be returned to the VP-3l Instructional System 


Department, for data summarization and analysis, by course 
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Subject Matter Experts (SMEs). On-site interviews will be 
conducted at sites selected by the Navy. 
3. Analysis Phase 
This phase addresses the following activities: 
(1) Summarization of collected data. 
(2) Analysis of collected data. 
(3) Revision procedures on the basis of analysis. 

Discrepancies in task skills or knowledges will 
require task analysis and identification of learning goals. 
Learning goals will be recorded on the Revision Specification 
Worksheet (Appendix E), placed in a learning heirarchy and 
assigned for authoring. The course syllabus will be modified 
as necessary. 

The Management Plan assigns (SMEs) the responsibility 
for summarizing and analyzing supervisor and course graduate 
questionnaire responses. The SMEs' duties consist of 
comparing fleet squadron input to input from other data 
sources. They then determine what action is to be taken to 
correct discrepancies. The summarized/analyzed data is then 
forwarded to the training officer/course supervisor for 
review. The Management Plan also requires a few specific 
steps to be carried out in the analysis data stage. These 
actions are discussed in Chapter Five. 

4. Revision Phase 
The revision phase is the responsibility of the ISD 


Department of the Navy. The instructional materials will be 
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revised in accordance with revision specifications from the 
analysis phase. According to the Management Plan, revisions, 
for the most part, will be dictated by the data collection 
and analysis results. SMEs and instructors will determine 
jointly, the nature and extent of the revisions to be made, 
but SMEs are ultimately responsible for instituting the 
actual changes. A few specific procedural details concerning 
revisions are presented in the Management Plan. These 


procedures will be discussed in Chapter Five. 


E. PERSONNEL AND SCHEDULING REQUIREMENTS 

The development and data collection phases were to be 
initiated by Courseware, oar and due to the time 
requirements for questionnaire distribution and on-site 
interviewing, the analysis and revision phases were to be 


conducted by the Navy. 
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IV. METHODOLOGY 


The data used in this thesis came from Uniformed Services 
publications, commercial books and reports, and various 
studies conducted by and for the Military Services. Dats 
data reflects the current design, implementation and 
Operation standards for United States Navy training programs. 
Additional background data was gathered through informal 
interviews with various VP FRS training administrators and 
managers. 

The intention of this thesis will be realized through the 
analysis of the P-3 Instructional System External Evaluation 
Plan. The Plan will be related to the MInterservice 
Procedures for Instructional System Development model 
external evaluation process as well as other standardized ISD 
concepts. From this analysis, a better understanding of the 
external evaluation plan will be gained, and recommendations 
for an improved evaluation process and overall training 
system will be made. The report will conclude with a 
discussion on the feasibility of implementing the Quality 


Control Plan. 
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V. ANALYSIS OF THE P-3 INSTRUCTIONAL SYSTEM EXTERNAL 


EVALUATION PLAN 


Previous sections of this thesis have related the 
importance of utilizing the ISD process to develop VP FRS 
aircrew training, how an evaluation process is essential to 
the creation of the closed-loop training system, and_ the 
development status for the P=-3 Instructional System Quality 
Gentrol Plan. This section will analyze the existing P-3 
Instructional System External Quality Control Plan. The 
author realizes that due to extenuating circumstances’ the 
plan, as it now stands, is not complete, continuous, of 
incorporated into the FRS training program. This analysis is 
being conducted to offer suggestions that should aid in the 
completion and implementation of the plan; thus resulting in 
an improved VP FRS aircrew training system. This analysis of 
the P-3 Instructional System External Evaluation Plan is not 
an attempt to segregate it from the remainder of the system, 
but is simply an attempt to focus on improving one aspect of 
the training program with the intention of strengthening the 
entire system. 

The procedure for the analysis will consist of comparing 
the external evaluation phase of the P=-3 Instructional System 
to the external evaluation phase of the IPISD model. The 


IPISD model was chosen as the comparison standard because it 
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provides a description of the approved techniques and 
procedures to be used for interservice training and includes 
requirements from all branches of the Armed Services. 
Additionally, Vineberg and Joyner [Ref. 7] found, in a study 
of all Armed Forces ISD models, that following external 
evaluation, the IPISD model provides the most guidance for 
isolating causes of performance discrepancies. The format 
for the comparison will consist of presenting a summary of 
each step of the IPISD Model external evaluation procedures 
immediately followed by a discussion concerning the P-3 
Instructional System External Quality ConErolL Plan 
accomplisments for that step and suggested recommendations 
for changes to the plan. 

The section will conclude with a discussion of the 
feasibility of implementing the external evaluation program, 
once it has been redesigned and can meet the appropriate ISD 


Standards. 


A. COMPARISON OF THE EXTERNAL EVALUATION PROCESSES 
il. Introde@etion 
The IPISD Model specifies that the primary purpose 
for conducting external evaluation is to ensure that course 
graduates can do the job for which they were trained, at the 
expected level of competency. External evaluation also 


checks to see if the job has changed since it was last 
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analyzed, thus, in a sense, providing a revalidation of the 
last task listing. Figure 5.1 illustrates the position 
(Block V.2) in the ISD cycle where external evaluation takes 
place. One must understand that although the evaluation is 
conducted after students have successfully completed 


instruction and reported to the job, it is still a part of 


the closed-loop process required in an ISD training program. 
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Source: Ref. 8: p. 63 
Figure 5.1. IPISD Model Block Diagram 
The IPISD Model requires the following sources of 
input data to conduct external evaluation: 
(1) All available data from internal evaluation. 


(2) The job analysis, task selection, job performance 
measures, and setting selection information developed in 


Phase One. 
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(3) Any documentation approved by higher authority 
which will change the requirements of the course. 


Procedurally correct and properly conducted external 
evaluation will either confirm that course graduates can do 
their job at required performance levels or that students 
cannot meet job performance standards and which tasks are 
causing this failure. The specific steps for planning and 
Carrying out external evaluation are shown in Figure 5.2. 

Discussion: Basically the IPISD Model and the P-3 
Instructional System External Quality Control Plan (QCP) seem 
to agree on their underlying philosophies for conducting 
external evaluation. Both models are primarily concerned 
with whether or not: (1) graduates can adequately handle the 
gels) for which they were trained, (2) the job task 
requirements have changed, and (3) course updates due to 
material revisions can be made. 

The QCP states in its introduction that specific 
procedures for external quality control will not be given due 
to contract time limitations. Therefore, an initial external 
evaluation of the P=-3 Instructional System was never 
conducted nor were procedures identified for ongoing 
evaluation and revision. It should be reemphasized that 
although these procedures were never devised, the QCP does 


concur that they are necessary. 
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Figure 5.2. External Evaluation Steps 
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The QCP design was hampered by several constraints. 
Although most of the constraints were primarily concerned 
with either the total initial evaluation effort, or internal 
evaluation, one constraint had a serious effect on the 
external evaluation program. The constraint of concern is a 
shortage of naval personnel resources. This lack of properly 
trained personnel imposes serious restrictions on successful 
evaluation planning, data collection, data analysis, and 
training program revision. 

Recommendations: The total benefit derived from an 
ISD structured training program cannot be realized without 
adequate personnel capable of conducting external evaluation. 
Sufficient quantities of knowledgeable personnel should be 
formed into a team that is capable of providing continuous 
evaluation. Optimally the team should be well versed in the 
area of ISD procedures and techniques, should have a_ good 
understanding of the job to be evaluated, and should not be 
connected to the training program. One possibility for 
forming such a team 1S to employ an instructional 
psychologist as team leader and use IS Department SMEsS/track 
Managers rotating out of VP-30 and VP-31 as the remaining 
members to form an independent evaluation unit. Another 
means of forming the evaluation team could be through 
utilization of personnel from agencies such as TAEG, NTEC, or 


Naval Instructional Technology Development Center (NITDC). 
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The problem with this sort of team would be their lack of 
Knowledge concerning the job which they are evaluating. 

The formation of an independent external evaluation 
team 1S a valuable part of the ISD concept. A study should 
be conducted immediately to determine how an effective and 
efficient evaluation team can be formed. 

2. Procedure 

The procedures for conducting external evaluation in 
accordance with the IPISD Model, Figure 5.2, are divided into 
three major sections: planning evaluation, collecting data, 
and consolidating the data and making recommendations. These 
three sections will be covered in detail. 

a. Plan Evaluation 

Planning evaluation is a process that is to be 
completed prior to beginning an external evaluation. During 
this process decisions are made to determine how’ the 
evaluation will be conducted by addressing four questions: 
who will provide data, what data is needed, exactly when the 
external evaluation will take place, and what methods will be 
used to gather the data? Since planning the evaluation is 
such an important part of the quality control concept, these 
four questions will be covered separately. 

(1) Determine Who Will Provide the Data. The 
IPISD Model specifies that in most cases data gathered for 


analysis will come from five major sources: 
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(l) Baseline data gathered prior to the development of 
instruction (see Figure 5.1, block II.2). 


(2) On-the-job course graduates. 
(3) Supervisors of on-the-job course graduates. 


(4) Evaluation team members (ideally composed of com- 
mand and school personnel). 


(5) Records dccumenting students' performance during 
instruction. 


Additionally, a check should be made on 
Students who are not assigned to the job for which they are 
trained. Find out if they were initially assigned to that 
job and why they have been reassigned. 

Discussion: The QCP specifies that data 
will be collected from both course graduates and supervisors 
of course graduates. However, the plan doesn't address 
utilizing data from the baseline study, evaluation team 
members, or records of students' performance. These three 
sources of information can provide invaluable information 
during external evaluation. Baseline data becomes a 
necessity if the technique of job performance evaluation is 
to be utilized to assess graduates' on-the-job performance, 
and evaluation team members' input data is extremely 
important no matter what technique is used to evaluate a 
graduate's ability to perform on the job. 

The construction of the P-3 Instructional 
System is such that records of students’ performance during 


instruction are available for use in evaluation. This sort 
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of data becomes especially important when the other sources 
of data provide conflicting conclusions. 

Recommendations: The QCP should state 
specifically and completely who will provide the data needed 
for evaluation. The author feels that the IPISD Model 
provides complete coverage of who will provide data. 
Therefore, the QPC would do well to adopt the data collecting 
procedures specified by the IPISD Model. 

(2) Determine What Data Are Required. The IPISD 
Model suggests that data gathered from graduates, supervisors 
of graduates, evaluation teams, and students' records be 
compared to the baseline data to obtain at least partial 
answers to questions such as: 

(1) Do graduates of the current training program show 
better performance on-the-job than graduates of past 
programs? 

(2) Do graduates of the current training program re- 
quire less on-the-job training than graduates of other 
courses? 

Additionally, the IPISD Model states that 
the data gathered from the four sources should be directed 
toward answering specific questions. The data gathered from 
graduates who are working on-the-job should provide answers 
to such questions as: 

(1) How well are they performing on the job? 


(2) What is the extent of training received since 
their arrival on-the-job? 
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get 


(3) How well did the course prepare them for the job? 


(4) What parts of the training program were relevant 
and irrelevant to the job? 


(5) In doing the job, how often do they use the skills 
taught? 


(6) In doing the job, what tasks do they find the most 
aa reTeg Lt? 


(7) In doing the job, for which tasks do they feel the 
least adequately prepared? 


(3) In doing the job, which tasks do they feel they 
verform the best? 


(9) How do they think the training program could be 
changed to better prepare them for the job? 


From supervisors of the working graduates, 


answers to such questions as: 
(£) How well are the graduates performing on-the-job? 


(2) How do the graduates compare to graduates of other 
courses, or those who received no training? 


(3) What type and how much on-the-job training have 
the graduates received? 


(4) In what areas are the graduates most adequately 
and inadequately prepared? 


(5) In what ways could the training program be im- 
proved? 


(6) Has the graduate had any problems operating equip- 
ment? 


(7) Has the graduate's performance been well above or 
well below the level expected? 


(8) Has the graduate been promoted or recommended for 
promotion? 


The evaluation team should attempt to get 


answers to such questions as: 
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(l) Did the graduate score well on the job performance 
measures? 


(2) Did the graduate fail any of the job performance 
measures? 


(3) How close do the job performance measures fit the 
actual job requirements? 


(4) Were guidelines followed to ensure that perfor- 
mance tests were properly administered and scored? 


(5) Is the job in question structured in accordance 
with regulations or approved doctrine? 


(6) Does the supervisor have satisfactory knowledge of 
the job? 


(7) Is any other information available that should be 
considered in making revision decisions? 


Discussion: Data gathering is the key 
element in ISD external evaluation. Therefore, any plan 
designed to conduct external evaluation must emphasize the 
collection of adequate amounts of data that are relevant to 
the questions for which answers are being sought. The IPISD 
Model states that ideally, external evaluation will be based 
Principally upon actual job performance as measured by JPMs 
that were produced during the design of ISD Model. The model 
goes on to say that often because of difficulties some data 
will have to be collected in another way. 

The P-3 Instructional System design concepts 
did not allow for the production of complete JPMs during its 
development. Therefore, a means to collect hard data for 


graduate on-the-job performance evaluation does not presently 


69 





exist. This Lornces tae OCP to rely primarily on 
questionnaires and interviews to gain the data necessary to 
answer the questions proposed above by the IPISD Model. 

Recommendations: The abscence of JPMs ina 
military service ISD based training program is not unusual. 
The advanced techniques, time, and manpower required to 
construct these measures often cause them to be omitted 
during the training program development. Additionally, 
utilization of the JPMs to gain accurate performance data for 
evaluation might be extremely desirable, but it is usually so 
expensive that it is not feasible. 

Well-designed questionnaires and oroperly 
conducted interviews are capable of producing a wealtn of 
information. t 1s recommended that if it is not feasible to 
construct JPMs at the present time, the QCP be structured to 
make optimum use of these alternate data gathering tools. 
Also orovisions should be made to allow the restructuring of 
the IPISD model provosed questions that deal with JPMs. 

(3) Determine When External Evaluation Will Take 
Place. Generally speaking, a course graduate should be given 
at least 30 days on the job before he is contacted. fis 
period of time will allow him to beccme accustomed to his new 
job and will give his supervisor encugh time to observe him 


and ovrovide a useful evaluation of his pberformance. 


70 





Do not allow more than 90 days to pass 
before contacting the graduate and his supervisor. Exceeding 
this length of time will make it difficult for the graduate 
to remember the particulars of his instruction and how well 
it prepared him for the job. Also, it will hamper the 
supervisor's ability to recall how well the graduate 
performed during the first few weeks on the job. 

Discussion: The QCP specifies that 
interviews be conducted with initial course graduates and 
their supervisors approximately six months after the graduate 
begins his operational assignment. The plan also requires 
that questionnaires be distributed and retrieved, but gives 
no timeframe for this to be accomplished. The author assumes 
that the questionnaires were to be distributed during the 
Same timeframe that the interviews were conducted (i.e., 
approximately six months after the graduate begins his job). 
Furthermore, since the plan only addresses interview times 
for initial course graduates, one must assume that the same 
relative timeframe would apply to graduates on a continuous 
basis. The general assumption that is being made is that 
the QCP, as it now stands, calls for external evaluation to 
take place approximately six months after a graduate begins 


his job in the fleet. 
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The six-month wait, before administering 
external evaluation as required by the QCP, certainly satis- 
fies the IPISD Model's minimum delay of 30 days. This delay 
allows the graduate time to get comfortable in the job and 
gives the supervisor a reasonable amount of time to observe 
him. On the other hand, the six-month wait far exceeds the 
IPISD Model's recommendation for a maximum delay of three 
months. This excessive delay would certainly make it diffi- 
cult for the graduate to recall details of his instruction 
and for his supervisor to pinpoint the graduates' initial 
capabilities. Additionally, the author feels that a six- 
month delay would impose a severe bias on the evaluation due 
to the strong influence of squadron training programs. 
During the author's three years in a VP fleet squadron, it 
was noted that except for rare instances, squadron training 
for a particular job began immediately upon assuming the 
duties of that job. Thus, after a six-month wait, a 
Graduate's performance on the job would be clouded by six 
months of intensive squadron training. 

Recommendations: External evaluation should 
be scheduled to take place during a timeframe that will 
minimize the degradations referred to in the discussion. 
Based upon the above information, it is suggested that’ the 
QCP adopt the IPISD Model's recommendation of conducting 


external evaluation between 30 days and 90 days after a 
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graduate reports to his fleet job. Due to the time normally 
required for cne to become "settled" into a new job, the 
optimum timeframe for the evaluation would probably be 60-90 
days after the graduate begins his job. 

(4) Determine How the Data Will be Gathered 

The three primary methods of gathering 
external evaluation data are: 

(1) Through job performance measures. 
(2) Through questionnaires. 
(3) Through personal interviews. 

The basic procedure is to provide question- 
naires to the maximum number of graduates and their supervi- 
sors and to pick a random sample of graduates and their 
Supervisors in the field for personal interviews. When a job 
performance evaluation is possible, it is usually subject 
mMatter/fevaluation experts who collect data from job perfor- 
mance measures and actual observation of the graduate on the 
job. 

Discussion: The QCP provides for external 
evaluation data to be gathered by questionnaires and personal 
interviews, which are two of the three methods mentioned in 
the IPISD Model. However, the QCP does not address data 
gathering via job performance measures, because as stated 
earlier, complete job performance measures were not construc- 


ted during the development of the P-3 Instructional System. 


73 





Specific procedures for determining how 
external evaluation data will be gathered are not covered in 
the QCP. Basically, the plan specifies that questionnaires 
will be sent to all initial course graduates and their 
Supervisors and that personal interviews will be conducted at 
Sites selected by the Navy. The P-3 Instructional System 
Management Plan makes additional provisions to provide 
questionnaires to all graduates which then makes the QCP 
process basically the same as the IPISD Model. 

The IPISD Model suggests that a random 
sample of graduates and their supervisors be picked for 
interviews. The QCP, as it presently exists, leaves this 
process up to the Navy. To maintain the least amount of bias 
and ensure an accurate sample population, an expert should 
probably be consulted to aid in selecting the sample. 

Recommendations: The QCP should be 
Structured to provide questionnaires to all course graduates. 
Additionally, the system should include measures ensuring a 
high return rate for the questionnaires. Procedures’ for 
selection of a sample population to be interviewed should oe 
devised by experts and structured such that Navy evaluation 
personnel could carry out the procedures on a continuous 


basis. 
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b. Collect Data 

Once a detailed external evaluation plan has been 
completed, procedures for gathering data become relatively 
Straight forward. The actual data collection falls into five 
major categories: (1) baseline data, (2) job performance 
evaluation data, (3) questionnaire data, (4) personal 
interview data, and (5) records of students' performance 
while enrolled in the training program. The procedures’ for 
data collection in each category are detailed below. 

(1) Collect Baseline Date. The recommended col- 
lection of baseline data takes place in Block II.2 as _ shown 
in Figure 5.1. The data is used both to confirm the need for 
the development of new instruction and for use in conducting 
external evaluation of the new instruction. Tostuimiwll. its 
purpose, the baseline study must satisfy three basic 
requirements. First, the data gathering devices (JPMs, 
etc.), must gather valid and reliable information. Second, 
he JPMS are administered, the sample group must be 
representative of the population to whom the results will 
apply. Third, to ensure accurate data gathering, the sample 
group must not be substantially different from the group 
involved in the external evaluation. 

Selecting an accurate sample population is 
fairly complex and possibly should be left to experts. The 


IPISD Model should be consulted for additional 





recommendations and guidelines on selecting sample sizes and 
sampling procedures, etc. 

Discussion: The IPISD Model relies on the 
utilization of previously constructed JPMs to perform its 
baseline study. The results of this study may then be used 
as data in the external evaluation process. Earlier in this 
section it was pointed out that due to a difference in design 
techniques, the Gevelopment of the P-3 Instructional System 
did not include the construction of complete job performance 
measures. Therefore, at the present time, a baseline study 
For the P-3 Instructional System, based upon JPMs does not 
and cannot exist. 

Recommendations: Baseline data plays a key 
role in external evaluation. The data producing study, if 
conducted properly, provides a standard to which the other 
sources of data may be compared. The ability to make this 
comparison provides a stable tool which tends to improve the 
worth of the evaluation. Therefore, it is recommended that 
steps be taken to determine what data exists within the P-3 
Instructional System that could be used in a baseline study. 
Measures should then be taken to produce formal baseline data 
Suitable for use in external evaluation. The study should 
adhere ee the sample population selection procedures 


discussed in the IPISD Model, and should contain background 
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data similar to that required in the ANALYZE JOB Block of the 
IPISD Model. 

(2) Collect Job Performance Evaluation Data. 
Block 1.3 of the IPISD Model (Figure 5.1) states that there 
must be some kind of JPM developed for every task selected 
EGr training. Without these valid JPMs, there will be no 
hard data (GL 4G no direct measure) Gr graduates' 
performance. Non-utilization of JPMs bases the external 
evaluation on interviews and impressionistic kinds of data 
which make decisions far riskier than those connected with 
JPMs. 

Soundly constructed JPMs can often be used 
to overcome the constraints of time, equipment, facility 
availability, and cost encountered in actual observation of 
graduates on-the-job, for external evaluation. When these 
constraints make it impractical to evaluate the actual 
cerformance of the tasks for which the graduates were 
trained, the evaluation can be based on the JPMs. 
Additionally, utilizing JPMs for evaluation allows them to be 
compared to the actual job requirements, thus allowing their 
revisions, when necessary, to ensure successful job 
performance. 

Discussion: The IPISD Model relies heavily 
upon JPM to provide a means of directly measuring a 


graduate's on-the-job performance. Miere 1s lier lLe doubt 
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that well constructed JPMs provide the optimum tool for data 
comparison, baseline studies, and even performance evaluation 
when actual observation of the graduate doing his job is not 
Practical. The lack of adequately constructed JPMs for each 
task in the P=-3 Instructional System make their use for 
conducting evaluation impossible at the present time. 
According to a P-3-Instructional System Evaluation Study 
conducted by Pacer Systems, Inc. for the Naval Training and 
Equipment Center ([Ref. 19], instructional system job 
performance measures are "Inadequately stated or not 
appropriate to tasks, or non-existent". 

To overcome the problem of inadequate and 
non-existent JPMs within the P-3 Instructional System, one of 
two possible steps must be taken. First, solidly constructed 
JPMs can be provided, or second, the QCP can rely upon the 
so-called "riskier" methods of gathering data for external 
evaluation. 

Recommendations: An immediate study should 
be conducted by qualified personnel to determine the exact 
Status of JPMs in the P-3 Instructional System. The study 
should focus upon the cost and time to provide adequate JPMs 
for each task selected for training within the system. The 
results of the study should then be compared to possible use 
of alternative methods of data gathering, such as 


questionnaires and personal interviews. The comparison should 
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concentrate on such areas as cost, shortest time to 
incorporate, accuracy of data, and long-term effectiveness of 
the data-gathering devices. 

(3) Collect Questionnaire Data. Although mailed 
questionnaires are not the most reliable method of gathering 
data from graduates and their supervisors, they are the least 
expensive. Meticulous attention to detail is required in 
questionnaire construction and utilization to ensure that 
sufficient quantities of accurate data are received for 
evaluation. When constructing the questionnaires, a few 
general rules should be Kept in mind: (1) ask specific 
questions, (2) list the tasks the graduate was trained to 
perform in the main body of the questionnaire, and (3) supply 
open-ended questions asking for suggestions on improving the 
training program. 

A large sample is required to provide 
accurate data. Therefore, a total population sampling is 
desirable to help ensure a large enough return rate _ to 
overcome variations in job requirements and erratic behavior 
of respondents. 

Discussion: The QCP states that question- 
naires are to be sent to all course graduates and their on- 
the-job supervisors. Sample questionnaires designed for this 
purpose are contained in Appendix A for the course graduate, 


and in Appendix C for his supervisor. 
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Recommendations: Properly constructed 
questionnaires utilized in a well-managed distribution and 
retrieval program can be instrumental in gathering accurate 
evaluation data in large quantities. Should the P-3 
Instructional System management personnel make the decision 
to use alternate data gathering methods instead of JPMs’ for 
their primary data collection tools, the quality and 
management plan for questionnaires will become critical. 
Therefore, it is recommended that the presently existing QCP 
Sample questionnaires be reviewed for construction accuracy 
and proper content. A questionnaire should then be devised 
for each aircrew position covered by the P-3 Instructional 
System. 

(4) Collect Personal Interview Data. Special- 
ists from the quality control team who are familiar with how 
the graduate was trained, are usually best suited to 
interview the graduate and his supervisor. Separate 
interviews with the selected sample populations of graduates 
and their supervisors should be structured to provide data on 
graduate assignments, utilization, and work proficiency. IE£ 
necessary, telephone interviews may be used. 

The interviews should consist of pre-planned 
lists of questions, such as those covered earlier in this 
section, used for the primary purpose of determining the 


graduate's proficiency. Also the interview should consider 
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how the graduates' skills are being utilized and how well the 
graduate is handling additional training. The structure of 
the interview should be kept flexible enough to fit the 
responses obtained from the person being interviewed. 

Discussion: Earlier in this section it was 
mentioned that the QCP provides for on-site interviews with 
course graduates and their on-the-job supervisors. Also the 
plan specifies that the sample to be interviewed should be 
selected by the Navy. The QCP does not provide any detailed 
procedures for selecting a sample population or for preparing 
and conducting interviews. 

The most useful information is usually 
obtained from an interview that is planned and conducted by 
an expert in the field. The interviewer should not only be 
well versed in interviewing techniques, but should also be 
familiar with the particular job held by the graduate he is 
interviewing. 

Once again it should be pointed out that if 
P-3 Instructional System Management Personnel choose methods 
for data gathering other than JPMs, then information from 
interviews will play a significant role in the external 
evaluation process. Therefore, construction of the interview 
structure, and the manner in which it is presented, should be 
accomplished in the most optimum fashion allowable, within 


imposed constraints. 
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Recommendations: Prior to the implementa- 
tion of the QCP, experts should be consulted to recommend a 
basic interview structure for each of the aircrew positions 
trained in the P-3 Instructional System. Additionally, 
training should be provided on interviewing techniques’ to 
those who will travel to the field to conduct the interviews. 
For example, the evaluation teams that were proposed in the 
introduction section of this chapter. 

The selection of sample populations’ and 
sampling procedures, as mentioned earlier, should probably be 
left to personnel knowledgeable in that field. Otherwise, 
bias and inaccurately gathered data would no doubt degrade 
the value of information gained from the interviews. 

(5) Obtain Records of Students' Performance 
During Instruction. Records of student's performance should 
be available to the external evaluator in case the other 
evaluation data indicate the students are not performing 
Satisfactorilly on the job. This data supplies information 
such as: (1) students' scores on each lesson post-test, (2) 
passing criteria for each test, and (3) the number of tries 
the students required to pass each test. The construction of 
the performance data sheet also allows students' scores to be 
easily compared in an effort to locate unfavorable trends. 

Discussion: The P-3 Instructional System 


Management Plan specifies that records of Students' 
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performance during training be maintained on "Student Test 
Score Transmittal Sheets". Shortly after implementation of 
their new ISD training program, VP-3l began utilizing a word 
processing system for data handling. This system has now 
replaced the "Student Test Score Transmittal Sheet" as a 
means of compiling and monitoring student's performance while 
under instruction. The word processor seems to be very 
efficient and accurate and should fit well into an external 
evaluation program. 

Recommendation: The use of a word processor 
has the capability OLE dramatically reducing the 
administrative workload for ISD personnel. There seems to be 
no reason to prevent this system from being used as an 
integral part of an external evaluation plan. When the QCP 
is being completed, the word processing system should be 
checked to ensure that the proper data are being gathered, 
and that the data are being provided in a format that will be 
usable in the evaluation plan. 

c. Consolidate Data and Make Recommendations 
The consolidation of data into a usable format is 
covered in Block I.1l of the IPISD Model. Using this format, 
graduate questionnaire data, supervisor questionnaire data, 
graduate interview data, supervisor interview data and JPMs 


may be compiled separately. Evaluation can then be initiated 
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by asking some pertinent questions, and checking the blocks 
of data for answers. 

Normally the first question to ask should be, can 
the job-holcing graduate perform the tasks he was trained to 
perform at the planned level of proficiency? This question 
usually leads to other questions such as: exactly which 
tasks are not being properly performed, and what does the job 
performance evaluation say about these unsatisfactorilly 
performed tasks? 

A few additional questions to ask are: 

(1) Are the tasks identified in the original job ana- 
lysis the same as those tasks the supervisors think are 


necessary to the job? 


(2) Are there areas where the graduates can score well 
on the JPMs but can't do the actual task? 


The above questions are provided to illustrate 
that there are no specific rules for evaluating data and 
making recommendations for change. Above all, common sense 
should be used when evaluating the data. If most graduates 
and supervisors are satisfied with the quality of the 
training program, very few, if any changes are usually 
necessary. On the other hand, if a significant number of the 
Sample population feels that training quality is low in some 
area, review the data to find possible causes for the 
inadequacy. The IPISD Model should be consulted for several 
more general suggestions for evaluating data and correcting 


problems. 
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AS a result of the above evaluation sechniques, 
changes may be made to the instructional system and/or the 
job structure. As an example of how the process works, if 
external evaluation shows problems with the instructional 
program, one would then go to Block V.3 REVISE SYSTEM, of the 
IPISD Model, and make the necessary changes. Next internal 
and external evaluation would be conducted again to check for 
errors. Eventually the oniy program revisions that will be 
necessary are those resulting from changes in content and 
doctrine. 

Once this point is reached, tne effectiveness of 
the training program should be checked by once again seexing 
answers to the questions: 

(1) Do graduates of the current training program show 
better overformance on the job than graduates of ovdast 
programs? 

(2) Do graduates of the current training program re- 
@uire less on-the-job training than graduates of other 
courses? 

To answer the first question, a sample from the 
baseline study that is as similar as possible to graduates of 
the instruction being evaluated is selected. The two groups 
are then compared on their performance on tne JPMs. If the 
performance level of the two groups is close (say 75 verses 
85 percent) experts should construct and administer a 


Statistical test to determine how often such a difference 


could be due to chance alone. The answer to the second 
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question can be obtained in a similar manner, but the data is 
gained from the questionnaires and interviews. 

Discussion: Following the data gathering 
effort, of external evaluation, are the extremely important 
steps of data consolidation and analysis. Together, data 
gathering and data analysis provide the basic framework for 
the entire concept of conducting external evaluation. 
Therefore, it is only proper that optimum analysis of the 
data be provided after extensive efforts have been expended 
to gather the "correct" data. 

Although the QCP does not specify adequate 
procedures for data consolidation and analysis, at the 
present time, it does provide some general guidelines. The 
plan places the responsibility for data analysis upon the 
Instructional System (IS) Department of VP-31l. The basic 
Process of data summarization and analysis within this 
department falls on the Subject Matter Experts (SMES), for 
each aircrew position. The basics for the SMEs analysis is a 
comparison of fleet input data to other data (i.e., data from 
Present students and instructors). The purpose of the 
analysis is to determine the performance of a graduate on- 
the-job, and to point out areas within the training program 
where the subject matter content is deficient. Once the SME 
has completed his analysis, he suggests corrective actions 


that will be considered by Training Officers/Course 
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Supervisors. Many of the problems which arise during 
analysis and revision attempts will be handled by a 
Curriculum Control Board (CCB), which is composed of 
representatives from the IS Department, FASO, VP-31 training 
office, etc. These procedures are not detailed enough to 
conduct an effective analysis, but they do seem to follow the 
basic recommendations of the IPISD Model. 

Recommendations: Regardless of whether external 
evaluation of the P-3 Instructional System is conducted by an 
evaluation team, as suggested by the author, or by SMEs from 
the IS Department, detailed administrative and analysis 
procedures must be provided if the evaluation is to be 
successful. The procedures must de explicit enough to 
overcome the high job turnover rate of professional 
evaluation team members and SMES, and must provide additional 
guidelines to compensate for non-professional evaluators 
(ie, SMEs who have not been extensively trained in 
interviewing, data interpretation, etc.) if they are used 
instead of an evaluation team. 

The process for completing the QCP should provide 
detailed procedures for questionnaire distribution, retrieval 
of all data, consolidation and analysis of data, and use of 
data for making instructional system revisions. The 
procedures must also take into account whether or not the use 


of  JPMs have been employed in the evaluation. Several of 
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these areas are addressed briefly in the IPISD Model, but 
much more detail must be provided if a realistic and accurate 
evaluation is expected to take place on a continuous basis. 
3. Outputs 
a. Products 
The primary product of the evaluation process is 
an external evaluation report (EXER). This report is composed 
of a summary statement of all external evaluation procedures, 
findings, interpretations, and recommendations for revisions. 
6b. Other Documentation 
This documentation includes supporting informa- 
tion such as: 


(1) A statement giving pertinent information about 
the graduate being evaluated. 


(2) The date evaluation took place, and the length of 
time between the completion of training and the conduction 
of external evaluation. 


(3) A statement describing now the evaluation was con- 
ducted. 


(4) A statement describing graduates' responses to 
questionnaires and interviews. 


(5) Job performance evaluation results. 


(6) How the evaluation team interpreted the data from 
items (3), (4) and (5) above. 


(7) Other information that influenced final recommen- 
dations. 


(8) Recommendation for revising the insructional 
system. 
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(9) Recommendations for revising the job structure. 
(10) General rcommendations not covered above. 

Discussion: Since the QCP has not been 
completed, it does not address the external evaluation report 
and its supporting information. However, the plan does 
recognize the need for such documentation, in that it 
discusses such products as a Revision Specification Worksheet 
and the necessity for course revisions. 

Recommendations: The IPISD Model seems to 
provide a complete list of external evaluation outputs. Itte 
is recommended that OCP designers refer to the IPISD Model 
for suggestions in completing this phase of the external 


evaluation plan. 


B. FEASIBILITY OF INCORPORATING THE P-3 INSTRUCTIONAL SYSTEM 
EXTERNAL QUALITY CONTROL PLAN 

The ISD based P=-3 Instructional System was implemented at 
VP-3l approximately three and one-half years ago. Since that 
time IS Department management and instructional personnel 
have succeeded in correcting almost all of the design errors 
and implementation problems associated with the new program. 
These corrections along with a series of continuous material 
and procedural improvements have allowed the system to evolve 


into a state of extreme stability. 
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The stable condition of the P-3 Instructional System plus 
the improved abilities of its managers and instructors, since 
implementation, suggest the present time as being favorable 
for incorporation of the evaluation phase of the program. 
This important step would provide the VP FRSs the "closed- 
loop" training system they initially sought to replace the 
traditional system. Since this thesis focuses on external 
evaluation, this discussion will be limited to that part of 
instructional system design and operation. 

Although this appears to be an opportune time to complete 
and incorporate the QCP, several obstacles exist that must be 
overcome or at least circumvented. One of the obstacles 
hindering successful implementation and overation of the QCP 
is the lack of enough adequately trained personnel to act as 
evaluators/administrators. No matter which evaluation 
concept is adopted (i.e., using an evaluation team or 
existing IS Department personnel for evaluators), there are 
not enough suitable personnel presently available to 
adequately do the job. This problem seems to have the 
highest probability of being solved by adopting the team 
concept. For example, an instructional design/evaluation 
expert could be hired on a permanent basis to provide team 
stability. The team members could then be obtained in the 
following manner. An individual is placed within the FRS as 


an SME/track manager for approximately one and one-half 
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years. Following this time period, he is then rotated to the 
external evaluation team for one and one-half years. This 
method would provide a Knowledgeable individual, for every 
aircrew position, to act as an external evaluator. The 
instructional design/evaluation expert and turn-over 
notebooks would be available to bring the newly installed 
evaluators "up-to-speed" on all procedures that deal with 
evaluation such as interviewing techniques, data analysis, 
etc. This suggestion for forming an evaluation team is based 
on the premise that experts have been hired to totally 
complete and aid in the initial implementation of the QCP. 
This concept could be duplicated if necessary to provide a 
complete team for each FRS. Depending upon the actual 
workload involved, the teams could possibly assist in 
conducting internal evaluation for the training system. This 
suggestion for an independent evaluation team could possibly 
eliminate the shortage of suitably trained personnel to 
conduct external evaluation. 

A second obstacle that must be faced in establishing a 
fully functional QCP is inadequate funding. With today's 
"hard-times™ economic situation, getting funding for anything 
short of an emergency situation is next to impossible. Yet, 
one must consider the benefits to be gained from a fully- 
implemented State-of-the-art Crawn2ng system Cie, 


continuous assurance that the training program is optimally 
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effective avd cost efficient). The actual cost for 
completion and implementation of the QCP appears to be very 
small when compared to the resultant benefits of such a plan. 
Perhaps consideration should be given to the IPISD Model 
Statement [Ref. 8] that implementation is an evolutionary 
process. It might be feasible to start out conducting 
partial external evaluation and work up to a full-fledged 
program after some period of time. According to the IPISD 
Model [Ref. 8], "...the spirit of the procedure is far more 
important than the letter”. 

A third obstacle that must be overcome is an inadequate 
management plan. The existing management plan was designed 
for use without a complete QCP incorporated. The IPISD Model 
Stresses the necessity of matching the management plan _ and 
the delivery system to lower internal costs. This applies to 
the evaluation process as well as the rest of the system. 
Therefore, the P-3 Instructional System Management Plan must 
be revised when a complete QCP is incorporated. 

The fourth and final area that will be addressed 
concerning incorporation of the QCP is the need for improved 
data handling capabilities. Once the FRSs begin conducting 
external evaluation, the IS Department must have a means to 
Store and utilize the additionally generated data. The 
solution to this problem could possibly be the Aviation 


Training Support System (ATSS) which is already partially in 
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existence at VP-3l1l's IS Department. Presently, the ATSS is 
incomplete and the existing software is very weak. However, 
it appears that acquiring the components to complete the 
system, and designing necessary software to handle the new 
demands imposed by the incorporation of the QCP could be both 


possible and practical. 
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VI. DISCUSSION AND SUMMARY OF RECOMMENDATIONS 


The P-3 Instructional System appears to be a well- 
constructed and expertly managed training program that 
produces a quality graduate who is thoroughly capable of 
doing his job in the fleet. The problem that exists is to 
prove this statement. Without a viable external evaluation 
plan, it is almost impossible to determine if the P-3 
Instructional System is superior to the traditional training 
program that it replaced and if the system is in fact 
producing the high caliber graduate needed to operate and 
maintain the sophisticated equipment on the P-3 aircraft. 

The intentions of this thesis have been to examine the 
Status of the P-3 Instructional System External Quality 
Cortrol Plan and to provide recommendations for tS 
completion and incorporation. The purpose for the 
examination of the Plan was to aid in the improvement of the 
instructional system and the quality of its graduates. The 
author realizes, as stated earlier, that the external 
evaluation plan cannot actually be thought of as ae single 
unit of the instructional system, but must be viewed as an 
integral part of the entire system. This idea was kept in 


mind during the analysis of the plan. 
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During the interim of this report, it was pointed out 
that due to certain unavoidable constraints, the external 
quality control plan was never completed. Although this fact 
is true, it is the author's opinion that Courseware, Inc. has 
provided a reasonably good start on the framework of the plan 
when compared to the IPISD Model. Additionally, the author 
believes that, due to the extreme importance of external 
evaluation and the long period of time that the Instructional 
System has been allowed to operate without it, experts should 
be employed immediately to complete the plan and aid in its 
incorporation. There is little doubt that this plan would 
more than pay for itself in the long run. 

During the analysis of the QCP, it was noted that neither 
ISD model provided detailed procedures for conducting 
external evaluation. The IPISD Model stated that this lack 
of detail was intentional and would allow each command to fit 
the model to its own particular instructional situation. 
This approach is probably a good idea, but the author 
strongly believes that the details for evaluation should be 
provided by experts and not Naval personnel who are 
unfamiliar with instructional design and evaluation. Rt Mies 
recommended that experts provide all procedural detail (1.e., 
general evaluation concepts, data collection, interviewing 
procedures, data analysis, etc.) and aid in the incorporation 


of the plan before turning the program over to the VP-31 IS 
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Department. This will provide a solid foundation for the 
evaluation and should minimize errors in design concepts and 
operating procedures. It should also be experts that decide 
whether it is feasible to construct and use JPMs’ for 
evaluation data, or whether the evaluation should be based on 
other data, such as questionnaires and interviews. 

Instructional Systems Development External Evaluation is 
a precise and time-consuming endeavor. Sinceeertt ismma 
possibility that completion and incorporation of the QCP may 
not be feasible at the present time, two alternative 
evaluation methods will be mentioned for consideration. 
First, Scriven [Ref. 20] proposed evaluation procedures whose 
purpose is based on "Goai-free evaluation". In other words, 
the evaluation examines the effects of an educational 
innovation and assesses the worth of the effects, whatever 
they are. With this method, the evaluator is not confined to 
the stated objectives, but is free to assess and evaluate 
Outcomes of any sort. This method is complex, but allows 
systematic judgment of appropriateness and general worth of 
the instructional product. The major emphasis of the model 
is verified performance. 

Second, Stufflebeam [Ref. 21] developed evaluation 
methods composed of a model called CIPP (Context, Input, 


Process, and Product). The model considers evaluation as a 
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continuous process whose purpose is to guide decision making. 
The major emphasis of the model is on continuous planning. 

Although the two evaluation models just discussed are 
widely accepted in the field of instructional program design, 
the author feels that, if at all possible, the P-3 
imstructional System should contain an incorporated 
evaluation plan that forms a closed-loop system. Using 
anything less than pure ISD techniques for evaluation would 
probably be a step backwards for the training system. 

The following recommendations for completing and 
incorporating the QCP are considered to be important: 

-- A team capable of providing continuous external eval- 
uation of the P=-3 Instructional System should be formed. 
Optimally the team should not be connected to the training 
program. 

-- A study should be conducted to determine whether the 
primary data source for the external evaluation should be 
based on job performance measures or alternate methods such 
as questionnaires and interviews. 

-- The QCP should require external evaluation to be con- 
ducted 30 to 90 days after a graduate reports to his job, as 
recommended by the IPISD Model. 

-- A study should be conducted to develop baseline data 
to be used as a standard for comparison in the external 


evaluation. 
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-- The currently existing QCP sample questionnaire for 
Graduates and their on-the-job supervisors should be reviewed 
for accuracy of construction and content. A questionnaire, 
for each aircrew position trained, should be developed based 
on the updated sample. 

-- Detailed interviewing procedures and correct and ac- 
curate methods for selecting sample populations should be 
provided by experts. 

-- Generally speaking, procedures for the entire evalu- 
ation process consisting of planning, data gathering, data 
analysis, and recommending revisions should be provided by 
experts. This provision will give the evaluation effort a 
solid foundation and pDrevent evaluation errors. 

-- The P-3 Instructional System Management Plan should 
be updated to deliver the new QCP. 

-- A study should be conducted to determine if the 
Aviation Training Support Systems (ATSS) is capable of 
handling the data generated by conducting external 
evaluation. 

The ISD based P-3 Instructional System represents a major 
step forward toward the application of "state-of-the-art" 
instructional design concepts. Continued refinement and an 
improved understanding of the ISD process should improve the 
instructional system by ensuring optimal training 


effectiveness, time efficiency and cost. 
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APPENDIX A 


INITIAL COURSE GRADUATE FOLLOW-UP QUESTIONNAIRE 
URGE 17s Dene 20) 


You are being asked to participate in the evaluation of 
the P-3 ISD training program. Please contribute your honest 
views on the training you received prior to your current 
assignment. All feedback will be considered in specifying 
revisions to the instructional materials. Your kelp is 


necessary for improving the training program for future 
students. 


Please complete the following information. All question- 
naire responses will be considered confidential. 


Name 
Rank 
Current Job Title 


Supervisor's Name 





Graduate of P-3 ISD Course Location 
Class No. 


1. How long have you been working at your present job? 
(Circle one) 


a- lk month 
b. 2 months 
c. 3-4 months 
d. 5-6 months 


2. How well do you feel you are able to perform your present 
jel? 


a. Above average 
b. Average 


c. Below average 


a2 





Have you received any on-the-job training for your 
present job since you arrived? 


a. Yes 
b. No 
If yes, please specify: 


How much? 


a ey 





What Kind? 








How well did the P-3 ISD training you received prepare 
you for your present job? (Circle one) 


a. More than adequate 
b. Adequate 

c. Inadequate 

d. Poor preparation 


Were vortions of the instruction irrelevant to your 
present job? 


a Yes 
b. No 


Please specify: 


In your present job, how often do you use the skills 
taught in P-3 ISD? 


a. Frequently 
b. Occasionally 


c. Not at all 
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Please specify if "not at all": 








In your present job, what tasks have given you the most 
Gist reuLlty? 


Please specify: 








What parts of the instruction do you think could be 
improved or added to better prepare future students’ for 
the job? 


Please specify: 


Other comments: 


BELL 





APPENDIX B 
INSTRUCTIONAL SYSTEMS SURVEY 
PILOT STUDENT 
(Ref. 18: p. H-26) 

Within the last six months you completed P-3 FRS 
training. You should have now had an opportunity to apply 
this training as a member of a flight crew. We would like 
you to think back to the training you received at the FRS and 
complete the attached questionnaire. The results of these 
questionnaires will be used to improve the training program 
for future students. Thank you for your cooperation in 
completing and returning this questionnaire within three 


weeks in the envelope provided. Your answers are IMPORTANT. 
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INITIAL COURSE GRADUATE FOLLOW-UP QUESTIONNAIRE 


Name 





Rank/Rate 

Aircrew Position/Designation 

Supervisor's Name 

Graduate of P-3 FRS Course Location 

Class No. 

You are being asked to participate in the evaluation of 
the P-$3 FRS training program. Please contribute your honest 
views on the FRS training you received prior to your current 
assignment. Your help is necessary for improving the 


training program for future students. 


Please complete the Following information. All 
questionnaire responses will be considered confidential. 


1. How long have you been assigned to your present crew 
position? 


b. 2 months 
c. 3-4 months 
ad. 5-6 months 


2. How well did the P-3 FRS training you received prepare 
you for your present aircrew assignment? 


a. More than adequate 
b. Adequate 
c. Inadequate 


If you answered c, please specify which areas: 
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Do you use the majority of the skills learned in the P-3 


FRS course on your present aircrew position? 
a. Frequently 


b. Occasionally 
c. Not at all 


Please specify those you do not use: 





In your present aircrew position, what tasks have given 


you the most difficulty? (Please specify): 


After having been on-the-job for a few months, 
the following subject areas covered in P-3 FRS 
would you like to see improved? ( ) 


a. systems 

Oye Normal Procedures 

S. Performance 

Sle Emergency Procedures 


which of 
training 


e. Instrument Procedures 

fies Safety/Survival/Landing Procedures 

ofc Navigation 

ite Oceanography 

i Search Tactics 

ie Localization Tactics 

ive Introduction to SS-3 (RADAR, ESM, IRDS, MAD) 
ae Active Tactics 

mM. Search/Kill Stores 

ne Plot Stabilization 

O« Communications 

oR Secondary Missions (Mining, SAR, 


Rigging, 


SOSUS, Soviet Naval Threat, RECCO, Coordinated 


Operations) 


Please specify the improvements you recommend: 
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Which of the following performance situations in the P-3 
FRS training would you like to see improved? (_) 


Ae 


lane 





Device Sessions 


Cockpit Procedural Trainers (CPT-Normal and 
Emergency Procedures) 

Flight Simulators (2F87-Normal and Emergency 
Procedures) 

Aircraft Labs (Preflight/Postflight) 


Fly Phase (I-vV) 
NAV Phase (I and NAV Extended) 
TAC Phase (I-IV) 


Please specify the improvements you recommend: 


Other comments: 
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APPENDIX C 


JOB SUPERVISOR FOLLOW-UP QUESTIONNAIRE 
(Ref. 17: p. 163) 


You are being asked to participate in the evaluation of 


the P=-3 ISD training program. Please contribute to this 
effort by evaluating the job performance of P-3 ISD initial 
course graduates. All feedback will be considered in 


specifying revisions to the instructional materials. Your 
help is necessary to improve the training for future course 
graduates. 


Please complete the Following information. All 
questionnaire responses will be considered confidential. 


Name 

Rank 

Job Title 

P-3 Course Graduates Supervised: 


Names: 





1. How long have you supervised the P-3 ISD initial course 
graduate(s)? (Circle one) 


a. 1 month 
b. 2 months 
c. 3-4 months 
d. 5-6 months 
2. How well are the graduates able to perform on the job? 
a. Above average 
b. Average 


c. Below average 
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Did the Graduate(s) receive any additional on-the-job 
training after arrival to the squadron? (Circle one) 


ae Yes 
b. No 
If yes, please specify: 


How much? 








What kind? 








How do these graduates perform on the job as compared to 
those who were trained in the past? 


a. Above average 
b. Average 
Cc. Below average 


Please specify if below average: 


In what task areaS were the graduates inadequately 
Drepared to perform on the job? 


Please specify: 


Has the graduate had accidents or been reprimanded for 
misuse or improper operation of equipment? 


a. Accident(s) 
b. Misuse of equipment 


c. Improper operation of equipment 
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Please specify: 








Has the graduate been warned for unusually bad 
performance? 


a. Yes 
b. No 


Please specify: 


Has the graduate been commended or recommended for 
promotion? 


a. Yes 
b. No 
c. Possible future consideration 


Other comments: 
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APPENDIX D 
INSTRUCTIONAL SYSTEMS SURVEY 
PILOT SUPERVISOR 
(Ref. 18: p. H-20) 

You are currently supervising the recent FRS graduate 
whose name appears on the enclosed questionnaire. You should 
have had some opportunity to observe his performance as a 
Flight crew member. We would like you to evaluate the 
graduate's performance by completing the questionnaire. The 
results of these questionnaires will be used to improve the 
training program. Thank you for your cooperation in 
completing and returning this questionnaire within three 


weeks in the envelope provided. Your answers are IMPORTANT. 
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the 


JOB SUPERVISOR FOLLOW-UP QUESTIONNAIRE 


Name 





Rank/Rate 

Crew Position/Designation 
P-3 FRS Course Name: 

FRS Graduate Name: 


You are being asked to participate in the evaluation of 
P—-3 FRS training program. Please contribute to this 


effort by evaluating the aircrew performance of P-3 FRS 
course graduates. Your help is necessary for improving the 
training program. 


Please complete the Following information. Ae 


questionnaire responses will be considered confidential. 


ilies 


How long have you supervised the FRS graduate names 
above? (Circle one) 


a. 1 month 
b. 2 months 
c. 3-4 months 
d. 5-6 months 


How well is the graduate able to perform his aircrew 
duties? (Please consider the following criteria in 
answering this question): (Circle one) 


- Qualified = Tasks performed in accordance with 
NATOPS or other established standards. 


Tasks NOT performed in accordance with 
NATOPS or other established standards 
when the performance does not meet the 
criteria specified for major errors. 


- Minor Errors 


- Major Errors = Tasks performed incorrectly AND which 
have the potential of: 
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1) Adversely affecting safety of flight 
Or personnel. 


2) Risking damage to equipment. 

3) Jeopardizing mission accomplishment. 
ae Qualified 
b. Minor errors 


G. Magor errors 


Please specify: 





Did the graduate require any additional training for 
Skills he should already have? (Circle one) 


ae Yes 
b. No 


If yes, please specify: 


Is the graduate making satisfactory progress towards 
designation as PP3P/PP2P/PPC? 


ae Yes 
b. No 


Please specify: 





Has the graduate had any accidents or incidents that were 
the result of personal error or been reprimanded for 
misuse or improper operation of equipment? 


a. Accident(s) or incident(s) 


b. Misuse of equipment 
c. Improper operation of equipment 


Lil 





Please specify: 








Has the graduate been warned for unusually deficient 
performance? 


ae Yes 
b. No 


Please specify: 


Has the graduate exhibited superior performance? 
a. Yes 


b. No 


Please specify: 


After having supervised the graduate "on-the-job" for a 
few months, in which of the following areas do you feel 
that this person requires more training? ( ) 


Ground School 


ae Systems 

eye Normal Procedures 

c. Performance 

d. Emergency Procedures 

e. Instrument Procedures 

ae Safety Survival/Landing Procedures 
g. Navigation 

Aire Oceanography 

ays Search Tactics 

Shc Localization Tactics 

Ke Introduction to SS-3 (RADAR, ESM, IRDS, MAD) 
alt. Active Tactics 

m. Search/Kill Stores 

ole Plot Stabilization 


daly2 


-_ 
aT 
iat iH 





On Communications 


Dis Secondary Missions (Mining, SAR, Rigging, 
SOSUS, Soviet Naval Threat, RCCO, Coordinated 
Operations) 


Device Sessions 


as Cockpit Procedural Trainers (CPT-Normal and 
Emergency Procedures) 

Bi Flight Simulators (2F87-Normal and Emergency 
Procedures) 

one Aircraft Labs (Preflight/Postflight) 

Flights 

a. Fly Phase (I-V) 

b. NAV Phase (I and NAV Extended) 

ey TAC Phase (I-IV) 


Please specify the improvements you recommend: 


Other comments: 


bi3 








Dale 


HEVISION SPECIFICATION WORKSIIEET 


APPENDIX & 


REVISION SPECIFICATION WORKSHEET 
(Ref. 17: p. 75) 
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